{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "li3F9kMOqZHz"
      },
      "source": [
        "<img src=\"../../../docs/images/DSPy8.png\" alt=\"DSPy7 Image\" height=\"150\"/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3wEDck3ZqZH0"
      },
      "source": [
        "# Using __<ins>M</ins>ulti-stage <ins>I</ins>nstruction <ins>P</ins>roposal & <ins>O</ins>ptimization (MIPROv2)__ in DSPy\n",
        "[![colab-badge](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/stanfordnlp/dspy/blob/main/examples/nli/scone/scone_with_MIPRO.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7DzBCQ0UqZH0"
      },
      "source": [
        "### FAQ 🙋\n",
        "#### 1) How does MIPRO work?\n",
        "At a high level, the MIPRO program optimizer works by first __proposing__ candidate fewshot example sets and instructions for each prompt in your program, and then __optimizing__ over these fewshot example sets and instructions as hyperparameters for a specified number of batches. Each batch, the optimizer evaluates different combinations of prompts on a subset of training inputs, which allows it to learn which combinations yield the best performance.\n",
        "\n",
        "#### 2) How much will MIPRO cost me to run?\n",
        "Note that __this notebook__ is free to run, because all LM calls have been cached. However, when using an optimizer on your own program, here is a breakdown of the upper bound of the number of calls to the task model and prompt model respectively:\n",
        "\n",
        "- **Task model calls**: MIPRO makes up to __O(TxPxM)__ task model calls if you run without minibatching, where T is the number of batches, P is the number of prompts in the program, and M is the size of the train set. This is because the model is evaluating the program on the train set each batch.  If you run **with minibatching** you can reduce calls even further to __O(TxPxB)__ where **B** is the minibatch size.  Note that every few steps (a parameter you set) MIPRO will also run a full eval over all **M** examples.\n",
        "\n",
        "- **Prompt model calls**: MIPRO makes up to N*P+10+(P+1) prompt model calls, where N is the number of instruction / fewshot example set candidates to generate for each prompt, and P is the number of prompts in the program. The extra 10 calls comes from generating a summary of the data in the training set, which we use in the meta prompt to create better instructions.  The extra (P+1) comes from program summarization where the proposer LLM will look at the program code and try to describe what each module does and what the whole program does.\n",
        "\n",
        "#### 3) How should I configure the hyperparameters?\n",
        "We have yet to run full hyperparameter sweeps with MIPRO, but based off of initial experimintation, we'd recommend the following:\n",
        "- __Batch num__: Gains can be seen after about 20-30 batches. However, 100-200 batches can help with adding on additional marginal gains.\n",
        "- __num candidates__: This hyperparameter controls the number of candidate prompts and fewshot example sets that are generated to optimize over. With more batches and less prompts to optimize, we can set n to be higher, as we have more batches to explore different combinations of prompts. If your program has between 2-3 modules and is the `num_batches=30`, we'd recommend ~`n=10`. If n is higher (say `n=100`), then we can go higher to ~`n=15`. If you have a program with only 1 module and are keeping the program 0-shot (ie. no fewshot examples), then `num_batches` should be set to equal `n`, because each batch can explore a new instruction.\n",
        "- __Training set size__: Between 100 and 500 training examples are recommended, however MIPROv2 can still work well with fewer. Increasing the training set size can help prevent overfitting, and only slightly increases cost for the full evaluation runs.\n",
        "\n",
        "#### 4) What should I do if I want to reduce the cost?\n",
        "You can always update hyperparameters accordingly, such as using a smaller train set, using less batches, or using a program with less modules.  You should take advantage of minibatching.\n",
        "Alternatively, one strategy would be to optimize using a cheaper task model (ie. locally hosted Llama-3), as initial experiments have shown that prompts optimized for a smaller model also transfer to working well on a larger model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SgTc-CutqZH1"
      },
      "source": [
        "### 0] Setup"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5Vo4Tb9srSow"
      },
      "source": [
        "First, we will install __DSPy__ if it's not there already. We'll also __load in the cached requests__ for this tasks, so that we don't actually need to call any LMs for this notebook. We'll also load in our pre optimized program from hugging face to inspect later."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JpijP_d7qZH2",
        "outputId": "c641a4b1-05f5-45cc-d715-4347c526b576"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/opt/miniconda3/envs/opt-prompt/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
            "  from .autonotebook import tqdm as notebook_tqdm\n"
          ]
        }
      ],
      "source": [
        "%load_ext autoreload\n",
        "%autoreload 2\n",
        "\n",
        "import sys\n",
        "import os\n",
        "\n",
        "try: # When on google Colab, let's clone the notebook so we download the cache.\n",
        "    import google.colab\n",
        "    repo_path = 'dspy'\n",
        "\n",
        "    !git -C $repo_path pull origin || git clone https://github.com/stanfordnlp/dspy $repo_path\n",
        "except:\n",
        "    repo_path = '.'\n",
        "\n",
        "if repo_path not in sys.path:\n",
        "    sys.path.append(repo_path)\n",
        "\n",
        "\n",
        "import pkg_resources # Install the package if it's not installed\n",
        "if not \"dspy-ai\" in {pkg.key for pkg in pkg_resources.working_set}:\n",
        "    !pip install -U pip\n",
        "    !pip install dspy-ai\n",
        "    !pip install openai~=1.12\n",
        "    !pip install -e $repo_path\n",
        "    !pip install --upgrade cloudpickle==3.0.0\n",
        "\n",
        "from huggingface_hub import hf_hub_download\n",
        "import zipfile\n",
        "import os\n",
        "\n",
        "repo_id = 'MichaelR207/MIPRO_notebook_cache_scone'\n",
        "cache_file_path = hf_hub_download(repo_id=repo_id, repo_type='dataset', filename='MIPRO_notebook_cache.zip')\n",
        "compiled_program_file_path = hf_hub_download(repo_id=repo_id, repo_type='dataset', filename='compiled_program.dspy')\n",
        "trial_logs_file_path = hf_hub_download(repo_id=repo_id, repo_type='dataset', filename='trial_logs.pickle')\n",
        "\n",
        "with zipfile.ZipFile(cache_file_path, 'r') as zip_ref:\n",
        "    zip_ref.extractall(\".\")\n",
        "\n",
        "os.environ[\"DSP_NOTEBOOK_CACHEDIR\"] = f\"{os.getcwd()}/MIPRO_notebook_cache\"\n",
        "\n",
        "import dspy\n",
        "import pandas as pd\n",
        "import glob\n",
        "import random\n",
        "from dspy.evaluate import Evaluate"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U-DaFCBvqZH2"
      },
      "source": [
        "We will also specify the __prompt LM model__ (in this case GPT 3.5), the __task LM model__ (Llama 3 8B)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "UHWzGRVgqZH2"
      },
      "outputs": [],
      "source": [
        "import openai\n",
        "\n",
        "### NOTE: if you'd like to run this code without a cache, you can remove these lines to configure your OPEN AI key ###\n",
        "# os.environ['OPENAI_API_KEY'] = \"TODO: ADD YOUR OPEN AI KEY HERE\"\n",
        "# openai.api_key = os.environ.get('OPENAI_API_KEY')\n",
        "# openai.api_base = \"https://api.openai.com/v1\"\n",
        "\n",
        "prompt_model_name = \"gpt-3.5-turbo-1106\"\n",
        "task_model_name = \"meta-llama/Meta-Llama-3-8B\"\n",
        "\n",
        "prompt_model = dspy.OpenAI(model=prompt_model_name, max_tokens=1000, stop=[\"\\n\\n\", \"\\n---\"])\n",
        "task_model = dspy.HFClientVLLM(\n",
        "    model=task_model_name,\n",
        "    port=7410,\n",
        "    url=[\"http://future-hgx-2:7500\", \"http://future-hgx-2:7501\", \"http://future-hgx-2:7502\", \"http://future-hgx-2:7503\", \"http://future-hgx-2:7504\", \"http://future-hgx-2:7505\", \"http://future-hgx-2:7506\", \"http://future-hgx-2:7507\"],\n",
        "    max_tokens=1000,\n",
        "    stop=[\"\\n\\n\", \"\\n---\", \"assistant\"],\n",
        ")\n",
        "\n",
        "dspy.settings.configure(lm=task_model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BFoPwDrUqZH2"
      },
      "source": [
        "### 1] Define Task"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s4Cyb1JtqZH2"
      },
      "source": [
        "Here, we'll define the program that we'd like to run, which is a multihop [...] (we can say that it was loosely inspired by a certain paper). We additionally load in the data, and define how we'd like to evaluate this task."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "fatal: destination path 'ScoNe' already exists and is not an empty directory.\n"
          ]
        }
      ],
      "source": [
        "!git clone https://github.com/selenashe/ScoNe.git"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {},
      "outputs": [],
      "source": [
        "class ScoNeSignature(dspy.Signature):\n",
        "    (\"\"\"context, question -> answer\"\"\")\n",
        "\n",
        "    context = dspy.InputField()\n",
        "    question = dspy.InputField()\n",
        "    answer = dspy.OutputField(desc=\"Yes or No\")\n",
        "\n",
        "class ScoNeCoT(dspy.Module):\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
        "\n",
        "    def forward(self, context, question):\n",
        "        return self.generate_answer(context=context, question=question)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 340,
          "referenced_widgets": [
            "e16044d880174c49af557bd12789493f",
            "531c380db44a404ebb168648ea77c3f4",
            "e8a46c6ac8a54fdbb44b2c8914552e64",
            "81cbe1842465400cba02a46309d98064",
            "f8ada809ecdb417ebc8214d860dd6552",
            "74d58314b1c449e0b7dec3bda8b653c2",
            "8b7b2b9489ae49049befae848d0fb1a1",
            "2d47a6a66054438d8e9a3c5e5767c056",
            "3b57d0a9768f4befb7509581289035ad",
            "134fa83980e142bf82d5b97cfc10da69",
            "35e4e99036894a2ba37d9ba23581a8ff",
            "835a1e675186490692e336da943d635d",
            "95bcef02eb794979b7873b81021e4b40",
            "d84324c4c3dc40fea04cc0df1539b4d8",
            "a6213bcdbcb24b1694204e6baeb763d8",
            "bc423d0878154adf940062660a8315ad",
            "0a1744c363f640b5b8939f32f6e72922",
            "bbd2db64988147f1a4f8856d890bb4c7",
            "ba2b4d7ecdbc4512acf180d8a0d602e8",
            "1bb155669e2a441d8992030e8aaa84a9",
            "43ed7af1d9c84ac8a6ec2195e48e60eb",
            "ac6822762e6b46bd862d6f923aeb4437",
            "348bb4fff900492cba8333156626a947",
            "697eaeb1a004493a9260a3a89671a9ca",
            "8a64950ea896468da3d10f46e9718ec8",
            "7b0b45020d0f45288e2f0e4ceff22524",
            "ad21e20d1f1b4b6ea74fdab03af8acdb",
            "1aab1d48c6124526bee6c74892ecd953",
            "598c6f37331e448889b175393a00deff",
            "20d8df0ee0a4442582686846757bcc7f",
            "fa4e50aaf53d4edfb9ca3046cbada2db",
            "96ad4454539145aea6549995344160e3",
            "acc3573e48f14dffb635f8632c6f6e74",
            "6c0432c6cd8f4cae9aeb8c2870b39922",
            "99b8f3882032404990f2b453111a63ba",
            "6c21df0657ce486381ba2310c7fa0029",
            "a20ac344a32340c785cd162fd0eb55b5",
            "6ba50c3ec9e542d5a8d2f900fbcb8689",
            "a298045042014c88939a9fe785fccda8",
            "84bed63c6597486ca6c39b9c0c4d20bd",
            "2a75d45acfb44463af974acb7a1b0d8e",
            "ae814b8e55454e0aadc48f7e595575ee",
            "ff4bcec9c18e4f04b1f7898daaffeb1a",
            "e9c64a790d684c9eaf7f49eea003f43c",
            "52f1714412df4ec78f6ff7d0f8a69862",
            "988055b44cf9411e8d45d8a4185fef07",
            "59be08d44c824d76b8525df14191d569",
            "6fde64f36dd84d8eae670efe95e2313a",
            "66e6502a463145daa440e967d8bdfdca",
            "29cab51fec0b4dacaa8f829ff217c839",
            "6ecbf23579324112981d4bce0c0ce369",
            "a105ef4f1f754c44b7bc0ec8edf0c0cc",
            "1c7d71e42c8c494f867b30d781beb681",
            "2f1e652cfa514054abfda794bdfa61a5",
            "b17f4731e8e44858889114c52b8f3dd4",
            "d50f5eafe90c4684bbdd96569b8ff247",
            "9f07fa213de247dabcd3f4213d35a97e",
            "782af098d37d4543a4a01ecfed4e5da2",
            "037b10adf9724e058c6468f4ca74ed86",
            "c7d10443100e49d28266aef9f644ed47",
            "d6a78aabcca74314a5b4bb98f350693a",
            "954f3cfcb5d44dddbf1d4db9e99c0d70",
            "62d8196169bb4a76a94c16d6f1ca61a6",
            "0a16c7f37b9a4bbe9bfe7e737ea62801",
            "942897aa22214745adee56d2c54447e9",
            "0b49910850f3445abe63b6bc7ac18412",
            "12a9763cd97742f4ab80c0494a398ca6",
            "9d4f14862b704d9bb53d9e74365d14ac",
            "56989c76e1534cfd8c9b0da93b3b8bf8",
            "f120c447645446e4a04791b97ce9ae93",
            "6a5d5ea44c0d4e4384b8956b48c34f14",
            "e3dd508496f44171b0d91aca0e8b16a5",
            "dc8077859eec4261a28d708ab06a1008",
            "9a733957f8fc446fbde053ba87289c71",
            "287c9f993cd44039923fdc122cd9e040",
            "5cfebfd6308349038f9cd7ad5bc00fe5",
            "56b80fc45dd247deadceffe17557f0f9",
            "7d80528c4b1c48318756230b0174923c",
            "398d7d36bbd041fe81ec633e973fc504",
            "d01ca0dd46ee4a4daeeee92214bc07d1",
            "6028a5e08f8641f5b8e8182e50f55419",
            "567646442eb940b09456328d49248945",
            "41961130caaf4537a4c1793574c785d7",
            "e9935b60c48d46469904492e20f9c2e8",
            "f048b50740e94bd8805c142eac1673b6",
            "a5effa5d67534fb4be2e3320c1fb2b9f",
            "09b3f2c456af41cba9264dbb9a724027",
            "e2f3a3377ad64a4cb9f3a42c1ac97344",
            "8e396041a4fd4e6db02410a09b7556c7",
            "4cd1dc71c80b401da19ed52ef5148d60",
            "1f23a95e292249a19055f70cda2622a3",
            "69412c7605ec48859baef6f31c91c520",
            "ee64a46b61454949ade4177c059bcf61",
            "de6507e9f45741218bf31a9af728b2bf",
            "38abb8f460d24c27a66c54bb0417f8ce",
            "33267de625db44c2a63d7c7d412a7e61",
            "19fe27a0df5a4d39873dd1031904405c",
            "dba7ce7c756d468b94d0bc8f4815ab6f",
            "926cbb119ea745cf9e344c0e50b75f62"
          ]
        },
        "id": "hiVgd3N7qZH3",
        "outputId": "81b97d83-7c5f-4763-d104-3ad320425a2a"
      },
      "outputs": [],
      "source": [
        "def load_scone(dirname):\n",
        "    dfs = []\n",
        "    filenames = ['one_not_scoped.csv', 'one_scoped.csv', 'no_negation.csv', 'one_scoped_one_not_scoped.csv', 'two_scoped.csv', 'two_not_scoped.csv']\n",
        "\n",
        "    for filename in filenames:\n",
        "        filename = os.path.join(dirname, filename)\n",
        "        df = pd.read_csv(filename, index_col=0)\n",
        "        df['category'] = os.path.basename(filename).replace(\".csv\", \"\")\n",
        "        dfs.append(df)\n",
        "    data_df = pd.concat(dfs)\n",
        "\n",
        "    def as_example(row):\n",
        "        # The 'one_scoped' file is from an earlier dataset, MoNLI, and\n",
        "        # so is formatted a bit differently:\n",
        "        suffix = '' if row['category'] == 'one_scoped' else '_edited'\n",
        "        # Reformat the hypothesis to be an embedded clause in a question:\n",
        "        hkey = 'sentence2' + suffix\n",
        "        question = row[hkey][0].lower() + row[hkey][1: ].strip(\".\")\n",
        "        question = f\"Can we logically conclude for sure that {question}?\"\n",
        "        # Binary task formulation:\n",
        "        label = \"Yes\" if row['gold_label' + suffix] == 'entailment' else \"No\"\n",
        "        return dspy.Example({\n",
        "            \"context\": row['sentence1' + suffix],\n",
        "            \"question\": question,\n",
        "            \"answer\": label,\n",
        "            \"category\": row['category'],\n",
        "        }).with_inputs(\"context\", \"question\")\n",
        "    return list(data_df.apply(as_example, axis=1).values)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Load and configure the datasets.\n",
        "all_train = load_scone(\"ScoNe/scone_nli/train\")\n",
        "\n",
        "random.seed(1)\n",
        "random.shuffle(all_train)\n",
        "\n",
        "# 1000 random train, 500 random dev:\n",
        "trainset, valset, testset = all_train[: 200], all_train[200: 400], all_train[400: 600]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Set up metrics\n",
        "NUM_THREADS = 10\n",
        "metric = dspy.evaluate.answer_exact_match\n",
        "\n",
        "kwargs = dict(num_threads=NUM_THREADS, display_progress=True)\n",
        "evaluate = Evaluate(devset=valset, metric=metric, **kwargs)\n",
        "\n",
        "program = ScoNeCoT()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hRGld1C1qZH3"
      },
      "source": [
        "### 2] Baseline Evaluation\n",
        "Now, we'll quickly evaluate our baseline program so that we can see how the performance using the Prompt Optimizer compares. We should see performance of about __58.0%__ on our trainset, __49.5%__ on our valset, and __55.0%__ on our testset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MU2aHQBTqZH3",
        "outputId": "786ba2f2-ae0a-4c68-b602-d601fb5a5aa5"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 116 / 200  (58.0): 100%|██████████| 200/200 [00:00<00:00, 907.05it/s]\n",
            "Average Metric: 99 / 200  (49.5): 100%|██████████| 200/200 [00:00<00:00, 862.68it/s]\n",
            "Average Metric: 110 / 200  (55.0): 100%|██████████| 200/200 [00:00<00:00, 647.97it/s]\n"
          ]
        }
      ],
      "source": [
        "baseline_train_score = evaluate(program,devset=trainset)\n",
        "baseline_eval_score = evaluate(program, devset=valset)\n",
        "baseline_test_score = evaluate(program, devset=testset)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cjCoL27yqZH3"
      },
      "source": [
        "### 3] Optimizing with MIPRO"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eEGOKjXprc7z"
      },
      "source": [
        "Now let's get into the key method in this notebook - optimizing our program with MIPRO!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G04L5j9iqZH3"
      },
      "source": [
        "#### 3a] Compile Program\n",
        "First, we'll get our optimized program. By default, we set `LOAD_PRECOMPILED_PROGRAM` to `True`, so that you can quickly access a program we've precompiled for you. However, if you wish to optimize yourself, `LOAD_PRECOMPILED_PROGRAM` can be set to `False` (though please note that this will require adding in your own LM API keys in the __Setup__ section above).\n",
        "\n",
        "MIPRO only needs a metric, DSPy module, and training set to see huge gains on your task!  You can instantiate a MIPRO Optimizer and compile in just two lines:\n",
        "```python\n",
        "teleprompter = MIPROv2(prompt_model=prompt_model, task_model=task_model, metric=metric, num_candidates=N, init_temperature=temperature)\n",
        "compiled_program = teleprompter.compile(program, trainset=trainset, valset=valset, num_batches=batches, max_bootstrapped_demos=1,max_labeled_demos=2, eval_kwargs=eval_kwargs)\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "NVfMJ_FpBlSI"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "/Users/michaelryan/Documents/School/Stanford/Research/dspy_official/dspy/examples/nli/scone/MIPRO_notebook_cache/compiler\n",
            "[Example({'context': 'The cowboy did not tell other people that he fall off a horse at the competition.', 'question': 'Can we logically conclude for sure that the cowboy did not tell other people that he fall off a bronco at the competition?', 'answer': 'No', 'category': 'one_not_scoped'}) (input_keys={'context', 'question'}), Example({'context': 'The woman not crying is not wearing rings.', 'question': 'Can we logically conclude for sure that the woman not crying is not wearing jewelry?', 'answer': 'No', 'category': 'one_scoped_one_not_scoped'}) (input_keys={'context', 'question'}), Example({'context': 'The people were not trying to keep their voices down so they woke a woman who is not indoors.', 'question': 'Can we logically conclude for sure that the people were not trying to keep their voices down so they woke a lady who is not indoors?', 'answer': 'No', 'category': 'two_not_scoped'}) (input_keys={'context', 'question'}), Example({'context': 'It is a lie that the man is not listening to bluegrass.', 'question': 'Can we logically conclude for sure that it is a lie that the man is not listening to music?', 'answer': 'Yes', 'category': 'two_scoped'}) (input_keys={'context', 'question'}), Example({'context': 'There is not a single person walking in the city.', 'question': 'Can we logically conclude for sure that there is not a single African walking in the city?', 'answer': 'Yes', 'category': 'one_scoped'}) (input_keys={'context', 'question'}), Example({'context': 'The man does not dress up when listening to music.', 'question': 'Can we logically conclude for sure that the man does not dress up when listening to bluegrass?', 'answer': 'No', 'category': 'one_not_scoped'}) (input_keys={'context', 'question'}), Example({'context': 'The man is not listening to opera.', 'question': 'Can we logically conclude for sure that the man is not listening to music?', 'answer': 'No', 'category': 'one_scoped'}) (input_keys={'context', 'question'}), Example({'context': \"There is a person walking in the city when it's not dark.\", 'question': \"Can we logically conclude for sure that there is a volunteer walking in the city when it's not dark?\", 'answer': 'No', 'category': 'one_not_scoped'}) (input_keys={'context', 'question'}), Example({'context': 'The girl who is not here is not wearing any jewelry at all.', 'question': 'Can we logically conclude for sure that the girl who is not here is not wearing any rings at all?', 'answer': 'Yes', 'category': 'one_scoped_one_not_scoped'}) (input_keys={'context', 'question'}), Example({'context': 'the boy, not girl, will play an piccolo, but not for another week', 'question': 'Can we logically conclude for sure that the boy, not girl, will play an instrument, but not for another week?', 'answer': 'Yes', 'category': 'two_not_scoped'}) (input_keys={'context', 'question'})]\n",
            "\u001b[93m\u001b[1mWARNING: Projected Language Model (LM) Calls\u001b[0m\n",
            "\n",
            "Please be advised that based on the parameters you have set, the maximum number of LM calls is projected as follows:\n",
            "\n",
            "\n",
            "\u001b[93m- Prompt Model: \u001b[94m\u001b[1m10\u001b[0m\u001b[93m data summarizer calls + \u001b[94m\u001b[1m10\u001b[0m\u001b[93m * \u001b[94m\u001b[1m1\u001b[0m\u001b[93m lm calls in program + (\u001b[94m\u001b[1m2\u001b[0m\u001b[93m) lm calls in program aware proposer = \u001b[94m\u001b[1m22\u001b[0m\u001b[93m prompt model calls\u001b[0m\n",
            "\u001b[93m- Task Model: \u001b[94m\u001b[1m25\u001b[0m\u001b[93m examples in minibatch * \u001b[94m\u001b[1m30\u001b[0m\u001b[93m batches + \u001b[94m\u001b[1m200\u001b[0m\u001b[93m examples in train set * \u001b[94m\u001b[1m3\u001b[0m\u001b[93m full evals = \u001b[94m\u001b[1m1350\u001b[0m\u001b[93m task model calls\u001b[0m\n",
            "\n",
            "\u001b[93m\u001b[1mEstimated Cost Calculation:\u001b[0m\n",
            "\n",
            "\u001b[93mTotal Cost = (Number of calls to task model * (Avg Input Token Length per Call * Task Model Price per Input Token + Avg Output Token Length per Call * Task Model Price per Output Token) \n",
            "            + (Number of calls to prompt model * (Avg Input Token Length per Call * Task Prompt Price per Input Token + Avg Output Token Length per Call * Prompt Model Price per Output Token).\u001b[0m\n",
            "\n",
            "For a preliminary estimate of potential costs, we recommend you perform your own calculations based on the task\n",
            "and prompt models you intend to use. If the projected costs exceed your budget or expectations, you may consider:\n",
            "\n",
            "\u001b[93m- Reducing the number of trials (`num_batches`), the size of the trainset, or the number of LM calls in your program.\u001b[0m\n",
            "\u001b[93m- Using a cheaper task model to optimize the prompt.\u001b[0m\n",
            "To proceed with the execution of this program, please confirm by typing \u001b[94m'y'\u001b[0m for yes or \u001b[94m'n'\u001b[0m for no.\n",
            "\n",
            "If you would like to bypass this confirmation step in future executions, set the \u001b[93m`requires_permission_to_run`\u001b[0m flag to \u001b[93m`False` when calling compile.\u001b[0m\n",
            "\n",
            "\u001b[93mAwaiting your input...\u001b[0m\n",
            "\n",
            "SOURCE CODE: ScoNeSignature(context, question -> answer\n",
            "    instructions='context, question -> answer'\n",
            "    context = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Context:', 'desc': '${context}'})\n",
            "    question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})\n",
            "    answer = Field(annotation=str required=True json_schema_extra={'desc': 'Yes or No', '__dspy_field_type': 'output', 'prefix': 'Answer:'})\n",
            ")\n",
            "\n",
            "class ScoNeCoT(dspy.Module):\n",
            "    def __init__(self):\n",
            "        super().__init__()\n",
            "        self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
            "\n",
            "    def forward(self, context, question):\n",
            "        return self.generate_answer(context=context, question=question)\n",
            "\n",
            "b: 10\n",
            "e 'NoneType' object has no attribute 'write'. using observations from past round for a summary.\n",
            "summary: Prediction(\n",
            "    summary='The dataset consists of examples that involve logical tests and negation related to language understanding and logical reasoning. Each example includes a context, a question, an answer, and a category label, and encompasses both single-scoped and two-scoped logical statements. The dataset seems to be focused on training models for handling negation in logical reasoning and has a broad range of potential applications.'\n",
            ")\n",
            "DATA SUMMARY: The dataset consists of examples that involve logical tests and negation related to language understanding and logical reasoning. Each example includes a context, a question, an answer, and a category label, and encompasses both single-scoped and two-scoped logical statements. The dataset seems to be focused on training models for handling negation in logical reasoning and has a broad range of potential applications.\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "  0%|          | 1/200 [00:00<00:00, 650.99it/s]\n",
            "  2%|▏         | 3/200 [00:00<00:00, 1005.35it/s]\n",
            "  0%|          | 1/200 [00:00<00:00, 750.19it/s]\n",
            "  0%|          | 1/200 [00:00<00:00, 686.69it/s]\n",
            "  0%|          | 1/200 [00:00<00:00, 488.96it/s]\n",
            "  1%|          | 2/200 [00:00<00:00, 738.24it/s]\n",
            "  0%|          | 1/200 [00:00<00:00, 762.74it/s]\n",
            "  0%|          | 1/200 [00:00<00:00, 409.00it/s]\n",
            "/opt/miniconda3/envs/opt-prompt/lib/python3.11/site-packages/optuna/samplers/_tpe/sampler.py:295: ExperimentalWarning: ``multivariate`` option is an experimental feature. The interface can change in the future.\n",
            "  warnings.warn(\n",
            "[I 2024-06-21 01:05:59,906] A new study created in memory with name: no-name-361c2f94-d354-463c-bbbe-c93d529a0192\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Using a randomly generated configuration for our grounded proposer.\n",
            "Selected tip: description\n",
            "PROGRAM DESCRIPTION: The program appears to be used to generate answers (Yes or No) based on a given context and question.\n",
            "task_demos \n",
            "\n",
            "\n",
            "\n",
            "Use the information below to learn about a task that we are trying to solve using calls to an LM, then generate a new instruction that will be used to prompt a Language Model to better solve the task.\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "DATASET SUMMARY: A description of the dataset that we are using.\n",
            "\n",
            "PROGRAM CODE: Language model program designed to solve a particular task.\n",
            "\n",
            "PROGRAM DESCRIPTION: Summary of the task the program is designed to solve, and how it goes about solving it.\n",
            "\n",
            "MODULE: The module to create an instruction for.\n",
            "\n",
            "TASK DEMO(S): Example inputs/outputs of our module.\n",
            "\n",
            "BASIC INSTRUCTION: Basic instruction.\n",
            "\n",
            "TIP: A suggestion for how to go about generating the new instruction.\n",
            "\n",
            "PROPOSED INSTRUCTION: Propose an instruction that will be used to prompt a Language Model to perform this task.\n",
            "\n",
            "---\n",
            "\n",
            "DATASET SUMMARY: The dataset consists of examples that involve logical tests and negation related to language understanding and logical reasoning. Each example includes a context, a question, an answer, and a category label, and encompasses both single-scoped and two-scoped logical statements. The dataset seems to be focused on training models for handling negation in logical reasoning and has a broad range of potential applications.\n",
            "\n",
            "PROGRAM CODE:\n",
            "ScoNeSignature(context, question -> answer\n",
            "    instructions='context, question -> answer'\n",
            "    context = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Context:', 'desc': '${context}'})\n",
            "    question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})\n",
            "    answer = Field(annotation=str required=True json_schema_extra={'desc': 'Yes or No', '__dspy_field_type': 'output', 'prefix': 'Answer:'})\n",
            ")\n",
            "\n",
            "class ScoNeCoT(dspy.Module):\n",
            "    def __init__(self):\n",
            "        super().__init__()\n",
            "        self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
            "\n",
            "    def forward(self, context, question):\n",
            "        return self.generate_answer(context=context, question=question)\n",
            "\n",
            "\n",
            "PROGRAM DESCRIPTION: The program appears to be used to generate answers (Yes or No) based on a given context and question.\n",
            "\n",
            "MODULE: self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
            "\n",
            "TASK DEMO(S): \n",
            "\n",
            "BASIC INSTRUCTION: context, question -> answer\n",
            "\n",
            "TIP: Make sure your instruction is very informative and descriptive.\n",
            "\n",
            "PROPOSED INSTRUCTION:\u001b[32m Provide a yes or no answer based on the given context and question. The context and question are used as input to generate the answer.\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "PROPOSED INSTRUCTION: Provide a yes or no answer based on the given context and question. The context and question are used as input to generate the answer.\n",
            "PROGRAM DESCRIPTION: The program appears to be used to generate answers (Yes or No) based on a given context and question.\n",
            "task_demos \n",
            "\n",
            "\n",
            "\n",
            "Use the information below to learn about a task that we are trying to solve using calls to an LM, then generate a new instruction that will be used to prompt a Language Model to better solve the task.\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "DATASET SUMMARY: A description of the dataset that we are using.\n",
            "\n",
            "PROGRAM CODE: Language model program designed to solve a particular task.\n",
            "\n",
            "PROGRAM DESCRIPTION: Summary of the task the program is designed to solve, and how it goes about solving it.\n",
            "\n",
            "MODULE: The module to create an instruction for.\n",
            "\n",
            "TASK DEMO(S): Example inputs/outputs of our module.\n",
            "\n",
            "BASIC INSTRUCTION: Basic instruction.\n",
            "\n",
            "TIP: A suggestion for how to go about generating the new instruction.\n",
            "\n",
            "PROPOSED INSTRUCTION: Propose an instruction that will be used to prompt a Language Model to perform this task.\n",
            "\n",
            "---\n",
            "\n",
            "DATASET SUMMARY: The dataset consists of examples that involve logical tests and negation related to language understanding and logical reasoning. Each example includes a context, a question, an answer, and a category label, and encompasses both single-scoped and two-scoped logical statements. The dataset seems to be focused on training models for handling negation in logical reasoning and has a broad range of potential applications.\n",
            "\n",
            "PROGRAM CODE:\n",
            "ScoNeSignature(context, question -> answer\n",
            "    instructions='context, question -> answer'\n",
            "    context = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Context:', 'desc': '${context}'})\n",
            "    question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})\n",
            "    answer = Field(annotation=str required=True json_schema_extra={'desc': 'Yes or No', '__dspy_field_type': 'output', 'prefix': 'Answer:'})\n",
            ")\n",
            "\n",
            "class ScoNeCoT(dspy.Module):\n",
            "    def __init__(self):\n",
            "        super().__init__()\n",
            "        self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
            "\n",
            "    def forward(self, context, question):\n",
            "        return self.generate_answer(context=context, question=question)\n",
            "\n",
            "\n",
            "PROGRAM DESCRIPTION: The program appears to be used to generate answers (Yes or No) based on a given context and question.\n",
            "\n",
            "MODULE: self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
            "\n",
            "TASK DEMO(S): \n",
            "\n",
            "BASIC INSTRUCTION: context, question -> answer\n",
            "\n",
            "TIP: Make sure your instruction is very informative and descriptive.\n",
            "\n",
            "PROPOSED INSTRUCTION:\u001b[32m Provide a yes or no answer based on the given context and question. The context and question are used as input to generate the answer.\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "PROPOSED INSTRUCTION: Provide a yes or no answer based on the given context and question. The context and question are used as input to generate the answer.\n",
            "PROGRAM DESCRIPTION: This program is designed to solve logical reasoning tasks by using language models. It takes in a context and a question as input, and then generates an answer based on the logical reasoning of the given context and question. The program uses a Chain of Thought model to process the input and produce the answer. The example provided uses the program to determine the logical conclusion of a given statement about people walking in the city, demonstrating its ability to reason and generate responses based on the input context and question.\n",
            "task_demos Context: It is not true that there is not a single person walking in the city.\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.\n",
            "Answer: No\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Use the information below to learn about a task that we are trying to solve using calls to an LM, then generate a new instruction that will be used to prompt a Language Model to better solve the task.\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "DATASET SUMMARY: A description of the dataset that we are using.\n",
            "\n",
            "PROGRAM CODE: Language model program designed to solve a particular task.\n",
            "\n",
            "PROGRAM DESCRIPTION: Summary of the task the program is designed to solve, and how it goes about solving it.\n",
            "\n",
            "MODULE: The module to create an instruction for.\n",
            "\n",
            "TASK DEMO(S): Example inputs/outputs of our module.\n",
            "\n",
            "BASIC INSTRUCTION: Basic instruction.\n",
            "\n",
            "TIP: A suggestion for how to go about generating the new instruction.\n",
            "\n",
            "PROPOSED INSTRUCTION: Propose an instruction that will be used to prompt a Language Model to perform this task.\n",
            "\n",
            "---\n",
            "\n",
            "DATASET SUMMARY: The dataset consists of examples that involve logical tests and negation related to language understanding and logical reasoning. Each example includes a context, a question, an answer, and a category label, and encompasses both single-scoped and two-scoped logical statements. The dataset seems to be focused on training models for handling negation in logical reasoning and has a broad range of potential applications.\n",
            "\n",
            "PROGRAM CODE:\n",
            "ScoNeSignature(context, question -> answer\n",
            "    instructions='context, question -> answer'\n",
            "    context = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Context:', 'desc': '${context}'})\n",
            "    question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})\n",
            "    answer = Field(annotation=str required=True json_schema_extra={'desc': 'Yes or No', '__dspy_field_type': 'output', 'prefix': 'Answer:'})\n",
            ")\n",
            "\n",
            "class ScoNeCoT(dspy.Module):\n",
            "    def __init__(self):\n",
            "        super().__init__()\n",
            "        self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
            "\n",
            "    def forward(self, context, question):\n",
            "        return self.generate_answer(context=context, question=question)\n",
            "\n",
            "\n",
            "PROGRAM DESCRIPTION: This program is designed to solve logical reasoning tasks by using language models. It takes in a context and a question as input, and then generates an answer based on the logical reasoning of the given context and question. The program uses a Chain of Thought model to process the input and produce the answer. The example provided uses the program to determine the logical conclusion of a given statement about people walking in the city, demonstrating its ability to reason and generate responses based on the input context and question.\n",
            "\n",
            "MODULE: self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
            "\n",
            "TASK DEMO(S):\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.\n",
            "Answer: No\n",
            "\n",
            "\n",
            "BASIC INSTRUCTION: context, question -> answer\n",
            "\n",
            "TIP: Make sure your instruction is very informative and descriptive.\n",
            "\n",
            "PROPOSED INSTRUCTION:\u001b[32m LogicalNegation(context: str, question: str -> answer: str) \"Given a context and a question involving negation, determine the logical conclusion and provide a yes or no answer based on the reasoning.\"\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "PROPOSED INSTRUCTION: LogicalNegation(context: str, question: str -> answer: str) \"Given a context and a question involving negation, determine the logical conclusion and provide a yes or no answer based on the reasoning.\n",
            "PROGRAM DESCRIPTION: This program appears to be designed to solve a task of logical reasoning and inference based on a given context and question. It uses a language model to generate an answer to the question based on the provided context. The program takes the context and question as input and returns a \"yes\" or \"no\" answer. In the example provided, the program uses the given context and question to produce a \"no\" answer, indicating that a logical conclusion cannot be made. The program seems to work by passing the context and question through a language model to generate the answer.\n",
            "task_demos Context: A man is not standing on top of a ladder that is leaned against a not so tall tree.\n",
            "Question: Can we logically conclude for sure that a man is not standing on top of a ladder that is leaned against a not so tall pine?\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the man is not standing on top of a ladder that is leaned against a not so tall tree. We can't conclude that the ladder is leaned against a pine tree, so we can't conclude that the man is not standing on top of a ladder that is leaned against a not so tall pine.\n",
            "Answer: No\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Use the information below to learn about a task that we are trying to solve using calls to an LM, then generate a new instruction that will be used to prompt a Language Model to better solve the task.\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "DATASET SUMMARY: A description of the dataset that we are using.\n",
            "\n",
            "PROGRAM CODE: Language model program designed to solve a particular task.\n",
            "\n",
            "PROGRAM DESCRIPTION: Summary of the task the program is designed to solve, and how it goes about solving it.\n",
            "\n",
            "MODULE: The module to create an instruction for.\n",
            "\n",
            "TASK DEMO(S): Example inputs/outputs of our module.\n",
            "\n",
            "BASIC INSTRUCTION: Basic instruction.\n",
            "\n",
            "TIP: A suggestion for how to go about generating the new instruction.\n",
            "\n",
            "PROPOSED INSTRUCTION: Propose an instruction that will be used to prompt a Language Model to perform this task.\n",
            "\n",
            "---\n",
            "\n",
            "DATASET SUMMARY: The dataset consists of examples that involve logical tests and negation related to language understanding and logical reasoning. Each example includes a context, a question, an answer, and a category label, and encompasses both single-scoped and two-scoped logical statements. The dataset seems to be focused on training models for handling negation in logical reasoning and has a broad range of potential applications.\n",
            "\n",
            "PROGRAM CODE:\n",
            "ScoNeSignature(context, question -> answer\n",
            "    instructions='context, question -> answer'\n",
            "    context = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Context:', 'desc': '${context}'})\n",
            "    question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})\n",
            "    answer = Field(annotation=str required=True json_schema_extra={'desc': 'Yes or No', '__dspy_field_type': 'output', 'prefix': 'Answer:'})\n",
            ")\n",
            "\n",
            "class ScoNeCoT(dspy.Module):\n",
            "    def __init__(self):\n",
            "        super().__init__()\n",
            "        self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
            "\n",
            "    def forward(self, context, question):\n",
            "        return self.generate_answer(context=context, question=question)\n",
            "\n",
            "\n",
            "PROGRAM DESCRIPTION: This program appears to be designed to solve a task of logical reasoning and inference based on a given context and question. It uses a language model to generate an answer to the question based on the provided context. The program takes the context and question as input and returns a \"yes\" or \"no\" answer. In the example provided, the program uses the given context and question to produce a \"no\" answer, indicating that a logical conclusion cannot be made. The program seems to work by passing the context and question through a language model to generate the answer.\n",
            "\n",
            "MODULE: self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
            "\n",
            "TASK DEMO(S):\n",
            "Context: A man is not standing on top of a ladder that is leaned against a not so tall tree.\n",
            "Question: Can we logically conclude for sure that a man is not standing on top of a ladder that is leaned against a not so tall pine?\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the man is not standing on top of a ladder that is leaned against a not so tall tree. We can't conclude that the ladder is leaned against a pine tree, so we can't conclude that the man is not standing on top of a ladder that is leaned against a not so tall pine.\n",
            "Answer: No\n",
            "\n",
            "\n",
            "BASIC INSTRUCTION: context, question -> answer\n",
            "\n",
            "TIP: Make sure your instruction is very informative and descriptive.\n",
            "\n",
            "PROPOSED INSTRUCTION:\u001b[32m Given a context and a question, use logical reasoning to determine if a logical conclusion can be made and generate a \"yes\" or \"no\" answer based on the provided information.\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "PROPOSED INSTRUCTION: Given a context and a question, use logical reasoning to determine if a logical conclusion can be made and generate a \"yes\" or \"no\" answer based on the provided information.\n",
            "PROGRAM DESCRIPTION: This program appears to be designed to solve tasks related to logical reasoning and inference based on a given context and question. The program takes a context and a question as input and uses a language model to generate an answer, which is either \"Yes\" or \"No\" based on logical inference. The program uses a Chain of Thought model to process the input and generate the answer. In the example provided, the program successfully processes the context and question to infer a logical conclusion and generate the answer \"Yes.\" Overall, the program is designed to use language models for logical reasoning tasks and inference.\n",
            "task_demos Context: The cowboy did not tell other people that he fall off a bronco at the competition.\n",
            "Question: Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the cowboy did not tell other people that he fall off a bronco at the competition. Bronco is a type of horse, so it is a type of horse that the cowboy fell off. Therefore, we can logically conclude that the cowboy did not tell other people that he fall off a horse at the competition.\n",
            "Answer: Yes\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Use the information below to learn about a task that we are trying to solve using calls to an LM, then generate a new instruction that will be used to prompt a Language Model to better solve the task.\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "DATASET SUMMARY: A description of the dataset that we are using.\n",
            "\n",
            "PROGRAM CODE: Language model program designed to solve a particular task.\n",
            "\n",
            "PROGRAM DESCRIPTION: Summary of the task the program is designed to solve, and how it goes about solving it.\n",
            "\n",
            "MODULE: The module to create an instruction for.\n",
            "\n",
            "TASK DEMO(S): Example inputs/outputs of our module.\n",
            "\n",
            "BASIC INSTRUCTION: Basic instruction.\n",
            "\n",
            "TIP: A suggestion for how to go about generating the new instruction.\n",
            "\n",
            "PROPOSED INSTRUCTION: Propose an instruction that will be used to prompt a Language Model to perform this task.\n",
            "\n",
            "---\n",
            "\n",
            "DATASET SUMMARY: The dataset consists of examples that involve logical tests and negation related to language understanding and logical reasoning. Each example includes a context, a question, an answer, and a category label, and encompasses both single-scoped and two-scoped logical statements. The dataset seems to be focused on training models for handling negation in logical reasoning and has a broad range of potential applications.\n",
            "\n",
            "PROGRAM CODE:\n",
            "ScoNeSignature(context, question -> answer\n",
            "    instructions='context, question -> answer'\n",
            "    context = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Context:', 'desc': '${context}'})\n",
            "    question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})\n",
            "    answer = Field(annotation=str required=True json_schema_extra={'desc': 'Yes or No', '__dspy_field_type': 'output', 'prefix': 'Answer:'})\n",
            ")\n",
            "\n",
            "class ScoNeCoT(dspy.Module):\n",
            "    def __init__(self):\n",
            "        super().__init__()\n",
            "        self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
            "\n",
            "    def forward(self, context, question):\n",
            "        return self.generate_answer(context=context, question=question)\n",
            "\n",
            "\n",
            "PROGRAM DESCRIPTION: This program appears to be designed to solve tasks related to logical reasoning and inference based on a given context and question. The program takes a context and a question as input and uses a language model to generate an answer, which is either \"Yes\" or \"No\" based on logical inference. The program uses a Chain of Thought model to process the input and generate the answer. In the example provided, the program successfully processes the context and question to infer a logical conclusion and generate the answer \"Yes.\" Overall, the program is designed to use language models for logical reasoning tasks and inference.\n",
            "\n",
            "MODULE: self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
            "\n",
            "TASK DEMO(S):\n",
            "Context: The cowboy did not tell other people that he fall off a bronco at the competition.\n",
            "Question: Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the cowboy did not tell other people that he fall off a bronco at the competition. Bronco is a type of horse, so it is a type of horse that the cowboy fell off. Therefore, we can logically conclude that the cowboy did not tell other people that he fall off a horse at the competition.\n",
            "Answer: Yes\n",
            "\n",
            "\n",
            "BASIC INSTRUCTION: context, question -> answer\n",
            "\n",
            "TIP: Make sure your instruction is very informative and descriptive.\n",
            "\n",
            "PROPOSED INSTRUCTION:\u001b[32m Please use the provided context and question to generate a logical inference for the answer using the language model. The answer should be \"Yes\" or \"No\" based on the logical reasoning and inference from the given context and question.\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "PROPOSED INSTRUCTION: Please use the provided context and question to generate a logical inference for the answer using the language model. The answer should be \"Yes\" or \"No\" based on the logical reasoning and inference from the given context and question.\n",
            "PROGRAM DESCRIPTION: The program is designed to solve a task of answering yes or no questions based on a given context. It appears to work by taking in a context and a question as inputs, and then using a language model to generate an answer to the question based on the provided context. In the example given, the program uses the provided context and question to generate a logical step-by-step reasoning and ultimately provide an answer of \"No\" to the question. This suggests that the program is designed to solve tasks that require logical reasoning and inference based on textual information.\n",
            "task_demos Context: It is not true that there is not a single person walking in the city.\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This does not necessarily mean that there is a celebrity walking in the city. There could be a non-celebrity person walking in the city.\n",
            "Answer: No\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Use the information below to learn about a task that we are trying to solve using calls to an LM, then generate a new instruction that will be used to prompt a Language Model to better solve the task.\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "DATASET SUMMARY: A description of the dataset that we are using.\n",
            "\n",
            "PROGRAM CODE: Language model program designed to solve a particular task.\n",
            "\n",
            "PROGRAM DESCRIPTION: Summary of the task the program is designed to solve, and how it goes about solving it.\n",
            "\n",
            "MODULE: The module to create an instruction for.\n",
            "\n",
            "TASK DEMO(S): Example inputs/outputs of our module.\n",
            "\n",
            "BASIC INSTRUCTION: Basic instruction.\n",
            "\n",
            "TIP: A suggestion for how to go about generating the new instruction.\n",
            "\n",
            "PROPOSED INSTRUCTION: Propose an instruction that will be used to prompt a Language Model to perform this task.\n",
            "\n",
            "---\n",
            "\n",
            "DATASET SUMMARY: The dataset consists of examples that involve logical tests and negation related to language understanding and logical reasoning. Each example includes a context, a question, an answer, and a category label, and encompasses both single-scoped and two-scoped logical statements. The dataset seems to be focused on training models for handling negation in logical reasoning and has a broad range of potential applications.\n",
            "\n",
            "PROGRAM CODE:\n",
            "ScoNeSignature(context, question -> answer\n",
            "    instructions='context, question -> answer'\n",
            "    context = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Context:', 'desc': '${context}'})\n",
            "    question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})\n",
            "    answer = Field(annotation=str required=True json_schema_extra={'desc': 'Yes or No', '__dspy_field_type': 'output', 'prefix': 'Answer:'})\n",
            ")\n",
            "\n",
            "class ScoNeCoT(dspy.Module):\n",
            "    def __init__(self):\n",
            "        super().__init__()\n",
            "        self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
            "\n",
            "    def forward(self, context, question):\n",
            "        return self.generate_answer(context=context, question=question)\n",
            "\n",
            "\n",
            "PROGRAM DESCRIPTION: The program is designed to solve a task of answering yes or no questions based on a given context. It appears to work by taking in a context and a question as inputs, and then using a language model to generate an answer to the question based on the provided context. In the example given, the program uses the provided context and question to generate a logical step-by-step reasoning and ultimately provide an answer of \"No\" to the question. This suggests that the program is designed to solve tasks that require logical reasoning and inference based on textual information.\n",
            "\n",
            "MODULE: self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
            "\n",
            "TASK DEMO(S):\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This does not necessarily mean that there is a celebrity walking in the city. There could be a non-celebrity person walking in the city.\n",
            "Answer: No\n",
            "\n",
            "\n",
            "BASIC INSTRUCTION: context, question -> answer\n",
            "\n",
            "TIP: Make sure your instruction is very informative and descriptive.\n",
            "\n",
            "PROPOSED INSTRUCTION:\u001b[32m Given the context and a question, use logical reasoning and inference to determine and provide a yes or no answer based on the information provided.\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "PROPOSED INSTRUCTION: Given the context and a question, use logical reasoning and inference to determine and provide a yes or no answer based on the information provided.\n",
            "PROGRAM DESCRIPTION: This program appears to be designed to solve contextual reasoning tasks by using language models. It takes in a context and a question, and uses a language model to generate a yes or no answer based on the given context and question. In the provided example, the program is used to determine if it can be logically concluded that there is a kayak nearby based on the given context. The program uses a chain of thought to process the input and generate the answer, taking into account the context and question provided. It uses the language model to reason through the context and question in order to provide the appropriate yes or no answer.\n",
            "task_demos Context: There is a boat nearby\n",
            "Question: Can we logically conclude for sure that there is a kayak nearby?\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that there is a boat nearby, but we don't know what type of boat it is. It could be a kayak, but it could also be a sailboat, a speedboat, or even a rowboat. We can't logically conclude that it is a kayak for sure.\n",
            "Answer: No\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Use the information below to learn about a task that we are trying to solve using calls to an LM, then generate a new instruction that will be used to prompt a Language Model to better solve the task.\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "DATASET SUMMARY: A description of the dataset that we are using.\n",
            "\n",
            "PROGRAM CODE: Language model program designed to solve a particular task.\n",
            "\n",
            "PROGRAM DESCRIPTION: Summary of the task the program is designed to solve, and how it goes about solving it.\n",
            "\n",
            "MODULE: The module to create an instruction for.\n",
            "\n",
            "TASK DEMO(S): Example inputs/outputs of our module.\n",
            "\n",
            "BASIC INSTRUCTION: Basic instruction.\n",
            "\n",
            "TIP: A suggestion for how to go about generating the new instruction.\n",
            "\n",
            "PROPOSED INSTRUCTION: Propose an instruction that will be used to prompt a Language Model to perform this task.\n",
            "\n",
            "---\n",
            "\n",
            "DATASET SUMMARY: The dataset consists of examples that involve logical tests and negation related to language understanding and logical reasoning. Each example includes a context, a question, an answer, and a category label, and encompasses both single-scoped and two-scoped logical statements. The dataset seems to be focused on training models for handling negation in logical reasoning and has a broad range of potential applications.\n",
            "\n",
            "PROGRAM CODE:\n",
            "ScoNeSignature(context, question -> answer\n",
            "    instructions='context, question -> answer'\n",
            "    context = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Context:', 'desc': '${context}'})\n",
            "    question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})\n",
            "    answer = Field(annotation=str required=True json_schema_extra={'desc': 'Yes or No', '__dspy_field_type': 'output', 'prefix': 'Answer:'})\n",
            ")\n",
            "\n",
            "class ScoNeCoT(dspy.Module):\n",
            "    def __init__(self):\n",
            "        super().__init__()\n",
            "        self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
            "\n",
            "    def forward(self, context, question):\n",
            "        return self.generate_answer(context=context, question=question)\n",
            "\n",
            "\n",
            "PROGRAM DESCRIPTION: This program appears to be designed to solve contextual reasoning tasks by using language models. It takes in a context and a question, and uses a language model to generate a yes or no answer based on the given context and question. In the provided example, the program is used to determine if it can be logically concluded that there is a kayak nearby based on the given context. The program uses a chain of thought to process the input and generate the answer, taking into account the context and question provided. It uses the language model to reason through the context and question in order to provide the appropriate yes or no answer.\n",
            "\n",
            "MODULE: self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
            "\n",
            "TASK DEMO(S):\n",
            "Context: There is a boat nearby\n",
            "Question: Can we logically conclude for sure that there is a kayak nearby?\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that there is a boat nearby, but we don't know what type of boat it is. It could be a kayak, but it could also be a sailboat, a speedboat, or even a rowboat. We can't logically conclude that it is a kayak for sure.\n",
            "Answer: No\n",
            "\n",
            "\n",
            "BASIC INSTRUCTION: context, question -> answer\n",
            "\n",
            "TIP: Make sure your instruction is very informative and descriptive.\n",
            "\n",
            "PROPOSED INSTRUCTION:\u001b[32m Determine if it can be logically concluded that a kayak is nearby based on the given context and question, and generate a yes or no answer using the language model.\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "PROPOSED INSTRUCTION: Determine if it can be logically concluded that a kayak is nearby based on the given context and question, and generate a yes or no answer using the language model.\n",
            "PROGRAM DESCRIPTION: This program is designed to solve tasks related to logical reasoning and inference based on a given context and question. It appears to work by taking in a context and a question as input, and then using a language model to generate an answer based on logical inference from the context. The `ScoNeSignature` function takes in the context and question as input, and returns an answer (either \"Yes\" or \"No) based on the logical reasoning process. The `ScoNeCoT` class sets up a chain of thought using the `ScoNeSignature` function to generate the answer based on the input context and question. The example provided demonstrates how the program uses the given context and question to generate a logical answer, \"Yes\".\n",
            "task_demos Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
            "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
            "Answer: Yes\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Use the information below to learn about a task that we are trying to solve using calls to an LM, then generate a new instruction that will be used to prompt a Language Model to better solve the task.\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "DATASET SUMMARY: A description of the dataset that we are using.\n",
            "\n",
            "PROGRAM CODE: Language model program designed to solve a particular task.\n",
            "\n",
            "PROGRAM DESCRIPTION: Summary of the task the program is designed to solve, and how it goes about solving it.\n",
            "\n",
            "MODULE: The module to create an instruction for.\n",
            "\n",
            "TASK DEMO(S): Example inputs/outputs of our module.\n",
            "\n",
            "BASIC INSTRUCTION: Basic instruction.\n",
            "\n",
            "TIP: A suggestion for how to go about generating the new instruction.\n",
            "\n",
            "PROPOSED INSTRUCTION: Propose an instruction that will be used to prompt a Language Model to perform this task.\n",
            "\n",
            "---\n",
            "\n",
            "DATASET SUMMARY: The dataset consists of examples that involve logical tests and negation related to language understanding and logical reasoning. Each example includes a context, a question, an answer, and a category label, and encompasses both single-scoped and two-scoped logical statements. The dataset seems to be focused on training models for handling negation in logical reasoning and has a broad range of potential applications.\n",
            "\n",
            "PROGRAM CODE:\n",
            "ScoNeSignature(context, question -> answer\n",
            "    instructions='context, question -> answer'\n",
            "    context = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Context:', 'desc': '${context}'})\n",
            "    question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})\n",
            "    answer = Field(annotation=str required=True json_schema_extra={'desc': 'Yes or No', '__dspy_field_type': 'output', 'prefix': 'Answer:'})\n",
            ")\n",
            "\n",
            "class ScoNeCoT(dspy.Module):\n",
            "    def __init__(self):\n",
            "        super().__init__()\n",
            "        self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
            "\n",
            "    def forward(self, context, question):\n",
            "        return self.generate_answer(context=context, question=question)\n",
            "\n",
            "\n",
            "PROGRAM DESCRIPTION: This program is designed to solve tasks related to logical reasoning and inference based on a given context and question. It appears to work by taking in a context and a question as input, and then using a language model to generate an answer based on logical inference from the context. The `ScoNeSignature` function takes in the context and question as input, and returns an answer (either \"Yes\" or \"No) based on the logical reasoning process. The `ScoNeCoT` class sets up a chain of thought using the `ScoNeSignature` function to generate the answer based on the input context and question. The example provided demonstrates how the program uses the given context and question to generate a logical answer, \"Yes\".\n",
            "\n",
            "MODULE: self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
            "\n",
            "TASK DEMO(S):\n",
            "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
            "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
            "Answer: Yes\n",
            "\n",
            "\n",
            "BASIC INSTRUCTION: context, question -> answer\n",
            "\n",
            "TIP: Make sure your instruction is very informative and descriptive.\n",
            "\n",
            "PROPOSED INSTRUCTION:\u001b[32m Please generate a logical answer (either \"Yes\" or \"No\") based on the given context and question using the `ScoNeSignature` function to perform logical reasoning and inference. Take the context and question as input to generate the answer.\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "PROPOSED INSTRUCTION: Please generate a logical answer (either \"Yes\" or \"No\") based on the given context and question using the `ScoNeSignature` function to perform logical reasoning and inference. Take the context and question as input to generate the answer.\n",
            "PROGRAM DESCRIPTION: This program appears to be designed to solve a task of answering yes or no questions based on a given context and question. The program takes in a context and a question as input and uses a language model to generate an answer. It seems to work by taking the input context and question, passing it through the ScoNeSignature function, and using the ChainOfThought method to generate the answer based on the input context and question. The example provided shows how the program can be used to answer a yes or no question based on a given context and question.\n",
            "task_demos Context: There is a athlete walking in the city when it's not dark.\n",
            "Question: Can we logically conclude for sure that there is a person walking in the city when it's not dark?\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the athlete is walking in the city, and we know that the athlete is a person. Therefore, we can logically conclude that there is a person walking in the city.\n",
            "Answer: Yes\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Use the information below to learn about a task that we are trying to solve using calls to an LM, then generate a new instruction that will be used to prompt a Language Model to better solve the task.\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "DATASET SUMMARY: A description of the dataset that we are using.\n",
            "\n",
            "PROGRAM CODE: Language model program designed to solve a particular task.\n",
            "\n",
            "PROGRAM DESCRIPTION: Summary of the task the program is designed to solve, and how it goes about solving it.\n",
            "\n",
            "MODULE: The module to create an instruction for.\n",
            "\n",
            "TASK DEMO(S): Example inputs/outputs of our module.\n",
            "\n",
            "BASIC INSTRUCTION: Basic instruction.\n",
            "\n",
            "TIP: A suggestion for how to go about generating the new instruction.\n",
            "\n",
            "PROPOSED INSTRUCTION: Propose an instruction that will be used to prompt a Language Model to perform this task.\n",
            "\n",
            "---\n",
            "\n",
            "DATASET SUMMARY: The dataset consists of examples that involve logical tests and negation related to language understanding and logical reasoning. Each example includes a context, a question, an answer, and a category label, and encompasses both single-scoped and two-scoped logical statements. The dataset seems to be focused on training models for handling negation in logical reasoning and has a broad range of potential applications.\n",
            "\n",
            "PROGRAM CODE:\n",
            "ScoNeSignature(context, question -> answer\n",
            "    instructions='context, question -> answer'\n",
            "    context = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Context:', 'desc': '${context}'})\n",
            "    question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})\n",
            "    answer = Field(annotation=str required=True json_schema_extra={'desc': 'Yes or No', '__dspy_field_type': 'output', 'prefix': 'Answer:'})\n",
            ")\n",
            "\n",
            "class ScoNeCoT(dspy.Module):\n",
            "    def __init__(self):\n",
            "        super().__init__()\n",
            "        self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
            "\n",
            "    def forward(self, context, question):\n",
            "        return self.generate_answer(context=context, question=question)\n",
            "\n",
            "\n",
            "PROGRAM DESCRIPTION: This program appears to be designed to solve a task of answering yes or no questions based on a given context and question. The program takes in a context and a question as input and uses a language model to generate an answer. It seems to work by taking the input context and question, passing it through the ScoNeSignature function, and using the ChainOfThought method to generate the answer based on the input context and question. The example provided shows how the program can be used to answer a yes or no question based on a given context and question.\n",
            "\n",
            "MODULE: self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
            "\n",
            "TASK DEMO(S):\n",
            "Context: There is a athlete walking in the city when it's not dark.\n",
            "Question: Can we logically conclude for sure that there is a person walking in the city when it's not dark?\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the athlete is walking in the city, and we know that the athlete is a person. Therefore, we can logically conclude that there is a person walking in the city.\n",
            "Answer: Yes\n",
            "\n",
            "\n",
            "BASIC INSTRUCTION: context, question -> answer\n",
            "\n",
            "TIP: Make sure your instruction is very informative and descriptive.\n",
            "\n",
            "PROPOSED INSTRUCTION:\u001b[32m Given a context and a question, use the provided context and question to determine if the logical conclusion is \"Yes\" or \"No\" and generate the corresponding answer.\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "PROPOSED INSTRUCTION: Given a context and a question, use the provided context and question to determine if the logical conclusion is \"Yes\" or \"No\" and generate the corresponding answer.\n",
            "PROGRAM DESCRIPTION: This program appears to be designed to solve tasks related to logical reasoning and inference based on given context and question. The program takes in a context and a question as input, and the goal is to produce a logical conclusion or answer based on the given context and question. It uses a language model to generate the answer by reasoning through the given context and question. The example provided demonstrates how the program works by taking in a context and a question and generating a logical answer based on the reasoning process.\n",
            "task_demos Context: A man is not holding anything in his hands.\n",
            "Question: Can we logically conclude for sure that a man is not holding tongues in his hands?\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that a man is not holding anything in his hands. This means that he is not holding any physical objects, including tongues. Tongues are not physical objects that can be held in one's hands. Therefore, we can logically conclude that a man is not holding tongues in his hands.\n",
            "Answer: Yes\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Use the information below to learn about a task that we are trying to solve using calls to an LM, then generate a new instruction that will be used to prompt a Language Model to better solve the task.\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "DATASET SUMMARY: A description of the dataset that we are using.\n",
            "\n",
            "PROGRAM CODE: Language model program designed to solve a particular task.\n",
            "\n",
            "PROGRAM DESCRIPTION: Summary of the task the program is designed to solve, and how it goes about solving it.\n",
            "\n",
            "MODULE: The module to create an instruction for.\n",
            "\n",
            "TASK DEMO(S): Example inputs/outputs of our module.\n",
            "\n",
            "BASIC INSTRUCTION: Basic instruction.\n",
            "\n",
            "TIP: A suggestion for how to go about generating the new instruction.\n",
            "\n",
            "PROPOSED INSTRUCTION: Propose an instruction that will be used to prompt a Language Model to perform this task.\n",
            "\n",
            "---\n",
            "\n",
            "DATASET SUMMARY: The dataset consists of examples that involve logical tests and negation related to language understanding and logical reasoning. Each example includes a context, a question, an answer, and a category label, and encompasses both single-scoped and two-scoped logical statements. The dataset seems to be focused on training models for handling negation in logical reasoning and has a broad range of potential applications.\n",
            "\n",
            "PROGRAM CODE:\n",
            "ScoNeSignature(context, question -> answer\n",
            "    instructions='context, question -> answer'\n",
            "    context = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Context:', 'desc': '${context}'})\n",
            "    question = Field(annotation=str required=True json_schema_extra={'__dspy_field_type': 'input', 'prefix': 'Question:', 'desc': '${question}'})\n",
            "    answer = Field(annotation=str required=True json_schema_extra={'desc': 'Yes or No', '__dspy_field_type': 'output', 'prefix': 'Answer:'})\n",
            ")\n",
            "\n",
            "class ScoNeCoT(dspy.Module):\n",
            "    def __init__(self):\n",
            "        super().__init__()\n",
            "        self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
            "\n",
            "    def forward(self, context, question):\n",
            "        return self.generate_answer(context=context, question=question)\n",
            "\n",
            "\n",
            "PROGRAM DESCRIPTION: This program appears to be designed to solve tasks related to logical reasoning and inference based on given context and question. The program takes in a context and a question as input, and the goal is to produce a logical conclusion or answer based on the given context and question. It uses a language model to generate the answer by reasoning through the given context and question. The example provided demonstrates how the program works by taking in a context and a question and generating a logical answer based on the reasoning process.\n",
            "\n",
            "MODULE: self.generate_answer = dspy.ChainOfThought(ScoNeSignature)\n",
            "\n",
            "TASK DEMO(S):\n",
            "Context: A man is not holding anything in his hands.\n",
            "Question: Can we logically conclude for sure that a man is not holding tongues in his hands?\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that a man is not holding anything in his hands. This means that he is not holding any physical objects, including tongues. Tongues are not physical objects that can be held in one's hands. Therefore, we can logically conclude that a man is not holding tongues in his hands.\n",
            "Answer: Yes\n",
            "\n",
            "\n",
            "BASIC INSTRUCTION: context, question -> answer\n",
            "\n",
            "TIP: Make sure your instruction is very informative and descriptive.\n",
            "\n",
            "PROPOSED INSTRUCTION:\u001b[32m Given the context and question, generate a logical answer based on the reasoning process.\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "PROPOSED INSTRUCTION: Given the context and question, generate a logical answer based on the reasoning process.\n",
            "CANDIDATE PROGRAM:\n",
            "Predictor 0\n",
            "i: Provide a yes or no answer based on the given context and question. The context and question are used as input to generate the answer.\n",
            "p: Answer:\n",
            "\n",
            "\n",
            "...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 13 / 25  (52.0): 100%|██████████| 25/25 [00:00<00:00, 654.31it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FULL TRACE\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 1 / 1  (100.0): 100%|██████████| 1/1 [00:00<00:00, 1175.20it/s]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "Provide a yes or no answer based on the given context and question. The context and question are used as input to generate the answer.\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.\n",
            "\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.\n",
            "\n",
            "Answer:\u001b[32m No\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Provide a yes or no answer based on the given context and question. The context and question are used as input to generate the answer.\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.\n",
            "\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.\n",
            "\n",
            "Answer:\u001b[32m No\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "...\n",
            "Score 52.0\n",
            "Best Combination: 1,{0: [[], [Example({'context': 'the boy, not girl, will play an horn, but not for another week', 'question': 'Can we logically conclude for sure that the boy, not girl, will play an instrument, but not for another week?', 'answer': 'Yes', 'category': 'two_not_scoped'}) (input_keys={'context', 'question'}), Example({'context': 'The person did not like raspberries.', 'question': 'Can we logically conclude for sure that the person did not like fruit?', 'answer': 'No', 'category': 'one_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'It is not true that there is not a single person walking in the city.', 'question': 'Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?', 'rationale': 'produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.', 'answer': 'No'}) (input_keys=None), Example({'context': 'It is not true that there is not a single person walking in the city.', 'question': 'Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?', 'answer': 'No', 'category': 'two_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'A man is not standing on top of a ladder that is leaned against a not so tall tree.', 'question': 'Can we logically conclude for sure that a man is not standing on top of a ladder that is leaned against a not so tall pine?', 'rationale': \"produce the answer. We know that the man is not standing on top of a ladder that is leaned against a not so tall tree. We can't conclude that the ladder is leaned against a pine tree, so we can't conclude that the man is not standing on top of a ladder that is leaned against a not so tall pine.\", 'answer': 'No'}) (input_keys=None), Example({'context': 'The people were outside not even trying to keep their voices down, and they woke up a mistress indoors.', 'question': 'Can we logically conclude for sure that the people were outside not even trying to keep their voices down, and they woke up a woman indoors?', 'answer': 'Yes', 'category': 'one_not_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'The cowboy did not tell other people that he fall off a bronco at the competition.', 'question': 'Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?', 'rationale': 'produce the answer. We know that the cowboy did not tell other people that he fall off a bronco at the competition. Bronco is a type of horse, so it is a type of horse that the cowboy fell off. Therefore, we can logically conclude that the cowboy did not tell other people that he fall off a horse at the competition.', 'answer': 'Yes'}) (input_keys=None), Example({'context': 'The diver has not seen any fish on his dive that did not go too deep.', 'question': 'Can we logically conclude for sure that the diver has not seen any tuna on his dive that did not go too deep?', 'answer': 'Yes', 'category': 'one_scoped_one_not_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'It is not true that there is not a single person walking in the city.', 'question': 'Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?', 'rationale': 'produce the answer. We know that it is not true that there is not a single person walking in the city. This does not necessarily mean that there is a celebrity walking in the city. There could be a non-celebrity person walking in the city.', 'answer': 'No'}) (input_keys=None), Example({'context': 'The cowboy did not tell other people that he fall off a bronco at the competition.', 'question': 'Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?', 'answer': 'Yes', 'category': 'one_not_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'There is a boat nearby', 'question': 'Can we logically conclude for sure that there is a kayak nearby?', 'rationale': \"produce the answer. We know that there is a boat nearby, but we don't know what type of boat it is. It could be a kayak, but it could also be a sailboat, a speedboat, or even a rowboat. We can't logically conclude that it is a kayak for sure.\", 'answer': 'No'}) (input_keys=None), Example({'context': 'the boy plays an instrument', 'question': 'Can we logically conclude for sure that the boy plays an flute?', 'answer': 'No', 'category': 'no_negation'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'The men were outside speaking loudly so as to wake the bridesmaid indoors', 'question': 'Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?', 'rationale': 'produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.', 'answer': 'Yes'}) (input_keys=None), Example({'context': 'There is not a single person not walking in the city.', 'question': 'Can we logically conclude for sure that there is not a single matriarch not walking in the city?', 'answer': 'Yes', 'category': 'one_scoped_one_not_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': \"There is a athlete walking in the city when it's not dark.\", 'question': \"Can we logically conclude for sure that there is a person walking in the city when it's not dark?\", 'rationale': 'produce the answer. We know that the athlete is walking in the city, and we know that the athlete is a person. Therefore, we can logically conclude that there is a person walking in the city.', 'answer': 'Yes'}) (input_keys=None), Example({'context': 'The man is not listening to techno.', 'question': 'Can we logically conclude for sure that the man is not listening to music?', 'answer': 'No', 'category': 'one_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'A man is not holding anything in his hands.', 'question': 'Can we logically conclude for sure that a man is not holding tongues in his hands?', 'rationale': \"produce the answer. We know that a man is not holding anything in his hands. This means that he is not holding any physical objects, including tongues. Tongues are not physical objects that can be held in one's hands. Therefore, we can logically conclude that a man is not holding tongues in his hands.\", 'answer': 'Yes'}) (input_keys=None), Example({'context': 'To believe that a man walks confidently not dropping produce is to believe a falsity.', 'question': 'Can we logically conclude for sure that to believe that a man walks confidently not dropping pears is to believe a falsity?', 'answer': 'No', 'category': 'two_scoped'}) (input_keys={'context', 'question'})]]} with Mean = 52.0\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n",
            "Average Metric: 113 / 200  (56.5): 100%|██████████| 200/200 [00:00<00:00, 587.61it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "UPDATING BEST SCORE WITH 56.5\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 98 / 200  (49.0): 100%|██████████| 200/200 [00:00<00:00, 660.76it/s]\n",
            "[I 2024-06-21 01:06:00,667] Trial 0 finished with value: 52.0 and parameters: {'0_predictor_instruction': 1, '0_predictor_demos': 2}. Best is trial 0 with value: 52.0.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "CANDIDATE PROGRAM:\n",
            "Predictor 0\n",
            "i: Determine if it can be logically concluded that a kayak is nearby based on the given context and question, and generate a yes or no answer using the language model.\n",
            "p: Answer:\n",
            "\n",
            "\n",
            "...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 11 / 25  (44.0): 100%|██████████| 25/25 [00:00<00:00, 712.57it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FULL TRACE\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 1 / 1  (100.0): 100%|██████████| 1/1 [00:00<00:00, 251.76it/s]\n",
            "[I 2024-06-21 01:06:00,762] Trial 1 finished with value: 44.0 and parameters: {'0_predictor_instruction': 6, '0_predictor_demos': 2}. Best is trial 0 with value: 52.0.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "Determine if it can be logically concluded that a kayak is nearby based on the given context and question, and generate a yes or no answer using the language model.\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.\n",
            "\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.\n",
            "\n",
            "Answer:\u001b[32m No\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Determine if it can be logically concluded that a kayak is nearby based on the given context and question, and generate a yes or no answer using the language model.\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.\n",
            "\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.\n",
            "\n",
            "Answer:\u001b[32m No\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "...\n",
            "Score 44.0\n",
            "CANDIDATE PROGRAM:\n",
            "Predictor 0\n",
            "i: Given a context and a question, use the provided context and question to determine if the logical conclusion is \"Yes\" or \"No\" and generate the corresponding answer.\n",
            "p: Answer:\n",
            "\n",
            "\n",
            "...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 14 / 25  (56.0): 100%|██████████| 25/25 [00:00<00:00, 500.19it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FULL TRACE\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 1 / 1  (100.0): 100%|██████████| 1/1 [00:00<00:00, 76.44it/s]\n",
            "[I 2024-06-21 01:06:00,877] Trial 2 finished with value: 56.0 and parameters: {'0_predictor_instruction': 8, '0_predictor_demos': 6}. Best is trial 2 with value: 56.0.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "Given a context and a question, use the provided context and question to determine if the logical conclusion is \"Yes\" or \"No\" and generate the corresponding answer.\n",
            "\n",
            "---\n",
            "\n",
            "Context: the boy plays an instrument\n",
            "Question: Can we logically conclude for sure that the boy plays an flute?\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: There is a boat nearby\n",
            "\n",
            "Question: Can we logically conclude for sure that there is a kayak nearby?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that there is a boat nearby, but we don't know what type of boat it is. It could be a kayak, but it could also be a sailboat, a speedboat, or even a rowboat. We can't logically conclude that it is a kayak for sure.\n",
            "\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city, but we don't know anything about the type of person. It could be a slaver, but it could also be a tourist, a local, or a homeless person. We can't logically conclude that it is a slaver for sure.\n",
            "\n",
            "Answer:\u001b[32m No\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Given a context and a question, use the provided context and question to determine if the logical conclusion is \"Yes\" or \"No\" and generate the corresponding answer.\n",
            "\n",
            "---\n",
            "\n",
            "Context: the boy plays an instrument\n",
            "Question: Can we logically conclude for sure that the boy plays an flute?\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: There is a boat nearby\n",
            "\n",
            "Question: Can we logically conclude for sure that there is a kayak nearby?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that there is a boat nearby, but we don't know what type of boat it is. It could be a kayak, but it could also be a sailboat, a speedboat, or even a rowboat. We can't logically conclude that it is a kayak for sure.\n",
            "\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city, but we don't know anything about the type of person. It could be a slaver, but it could also be a tourist, a local, or a homeless person. We can't logically conclude that it is a slaver for sure.\n",
            "\n",
            "Answer:\u001b[32m No\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "...\n",
            "Score 56.0\n",
            "CANDIDATE PROGRAM:\n",
            "Predictor 0\n",
            "i: Please use the provided context and question to generate a logical inference for the answer using the language model. The answer should be \"Yes\" or \"No\" based on the logical reasoning and inference from the given context and question.\n",
            "p: Answer:\n",
            "\n",
            "\n",
            "...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 17 / 25  (68.0): 100%|██████████| 25/25 [00:00<00:00, 1802.73it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FULL TRACE\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 1 / 1  (100.0): 100%|██████████| 1/1 [00:00<00:00, 71.35it/s]\n",
            "[I 2024-06-21 01:06:01,019] Trial 3 finished with value: 68.0 and parameters: {'0_predictor_instruction': 4, '0_predictor_demos': 5}. Best is trial 3 with value: 68.0.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "Please use the provided context and question to generate a logical inference for the answer using the language model. The answer should be \"Yes\" or \"No\" based on the logical reasoning and inference from the given context and question.\n",
            "\n",
            "---\n",
            "\n",
            "Context: The cowboy did not tell other people that he fall off a bronco at the competition.\n",
            "Question: Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This does not necessarily mean that there is a celebrity walking in the city. There could be a non-celebrity person walking in the city.\n",
            "\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This does not necessarily mean that there is a slaver walking in the city. There could be a non-slaver person walking in the city. Additionally, slavery is illegal in most places, so it is highly unlikely that there would be a slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m No\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Please use the provided context and question to generate a logical inference for the answer using the language model. The answer should be \"Yes\" or \"No\" based on the logical reasoning and inference from the given context and question.\n",
            "\n",
            "---\n",
            "\n",
            "Context: The cowboy did not tell other people that he fall off a bronco at the competition.\n",
            "Question: Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This does not necessarily mean that there is a celebrity walking in the city. There could be a non-celebrity person walking in the city.\n",
            "\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This does not necessarily mean that there is a slaver walking in the city. There could be a non-slaver person walking in the city. Additionally, slavery is illegal in most places, so it is highly unlikely that there would be a slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m No\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "...\n",
            "Score 68.0\n",
            "CANDIDATE PROGRAM:\n",
            "Predictor 0\n",
            "i: Given a context and a question, use logical reasoning to determine if a logical conclusion can be made and generate a \"yes\" or \"no\" answer based on the provided information.\n",
            "p: Answer:\n",
            "\n",
            "\n",
            "...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 14 / 25  (56.0): 100%|██████████| 25/25 [00:00<00:00, 1270.00it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FULL TRACE\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 1 / 1  (100.0): 100%|██████████| 1/1 [00:00<00:00, 280.50it/s]\n",
            "[I 2024-06-21 01:06:01,124] Trial 4 finished with value: 56.0 and parameters: {'0_predictor_instruction': 3, '0_predictor_demos': 8}. Best is trial 3 with value: 68.0.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "Given a context and a question, use logical reasoning to determine if a logical conclusion can be made and generate a \"yes\" or \"no\" answer based on the provided information.\n",
            "\n",
            "---\n",
            "\n",
            "Context: The man is not listening to techno.\n",
            "Question: Can we logically conclude for sure that the man is not listening to music?\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: There is a athlete walking in the city when it's not dark.\n",
            "\n",
            "Question: Can we logically conclude for sure that there is a person walking in the city when it's not dark?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the athlete is walking in the city, and we know that the athlete is a person. Therefore, we can logically conclude that there is a person walking in the city.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. However, we cannot logically conclude that it is not true that there is not a single slaver walking in the city, because we have no information about the presence of slavers in the city.\n",
            "\n",
            "Answer:\u001b[32m No\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Given a context and a question, use logical reasoning to determine if a logical conclusion can be made and generate a \"yes\" or \"no\" answer based on the provided information.\n",
            "\n",
            "---\n",
            "\n",
            "Context: The man is not listening to techno.\n",
            "Question: Can we logically conclude for sure that the man is not listening to music?\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: There is a athlete walking in the city when it's not dark.\n",
            "\n",
            "Question: Can we logically conclude for sure that there is a person walking in the city when it's not dark?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the athlete is walking in the city, and we know that the athlete is a person. Therefore, we can logically conclude that there is a person walking in the city.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. However, we cannot logically conclude that it is not true that there is not a single slaver walking in the city, because we have no information about the presence of slavers in the city.\n",
            "\n",
            "Answer:\u001b[32m No\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "...\n",
            "Score 56.0\n",
            "CANDIDATE PROGRAM:\n",
            "Predictor 0\n",
            "i: LogicalNegation(context: str, question: str -> answer: str) \"Given a context and a question involving negation, determine the logical conclusion and provide a yes or no answer based on the reasoning.\n",
            "p: Answer:\n",
            "\n",
            "\n",
            "...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 18 / 25  (72.0): 100%|██████████| 25/25 [00:00<00:00, 903.04it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FULL TRACE\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 1 / 1  (100.0): 100%|██████████| 1/1 [00:00<00:00, 139.84it/s]\n",
            "[I 2024-06-21 01:06:01,205] Trial 5 finished with value: 72.0 and parameters: {'0_predictor_instruction': 2, '0_predictor_demos': 3}. Best is trial 5 with value: 72.0.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "LogicalNegation(context: str, question: str -> answer: str) \"Given a context and a question involving negation, determine the logical conclusion and provide a yes or no answer based on the reasoning.\n",
            "\n",
            "---\n",
            "\n",
            "Context: The people were outside not even trying to keep their voices down, and they woke up a mistress indoors.\n",
            "Question: Can we logically conclude for sure that the people were outside not even trying to keep their voices down, and they woke up a woman indoors?\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: A man is not standing on top of a ladder that is leaned against a not so tall tree.\n",
            "\n",
            "Question: Can we logically conclude for sure that a man is not standing on top of a ladder that is leaned against a not so tall pine?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the man is not standing on top of a ladder that is leaned against a not so tall tree. We can't conclude that the ladder is leaned against a pine tree, so we can't conclude that the man is not standing on top of a ladder that is leaned against a not so tall pine.\n",
            "\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there is at least one person walking in the city. We can't conclude that this person is a slaver, so we can't conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m No\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "LogicalNegation(context: str, question: str -> answer: str) \"Given a context and a question involving negation, determine the logical conclusion and provide a yes or no answer based on the reasoning.\n",
            "\n",
            "---\n",
            "\n",
            "Context: The people were outside not even trying to keep their voices down, and they woke up a mistress indoors.\n",
            "Question: Can we logically conclude for sure that the people were outside not even trying to keep their voices down, and they woke up a woman indoors?\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: A man is not standing on top of a ladder that is leaned against a not so tall tree.\n",
            "\n",
            "Question: Can we logically conclude for sure that a man is not standing on top of a ladder that is leaned against a not so tall pine?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the man is not standing on top of a ladder that is leaned against a not so tall tree. We can't conclude that the ladder is leaned against a pine tree, so we can't conclude that the man is not standing on top of a ladder that is leaned against a not so tall pine.\n",
            "\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there is at least one person walking in the city. We can't conclude that this person is a slaver, so we can't conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m No\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "...\n",
            "Score 72.0\n",
            "CANDIDATE PROGRAM:\n",
            "Predictor 0\n",
            "i: Given the context and question, generate a logical answer based on the reasoning process.\n",
            "p: Answer:\n",
            "\n",
            "\n",
            "...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 14 / 25  (56.0): 100%|██████████| 25/25 [00:00<00:00, 938.26it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FULL TRACE\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 1 / 1  (100.0): 100%|██████████| 1/1 [00:00<00:00, 282.88it/s]\n",
            "[I 2024-06-21 01:06:01,283] Trial 6 finished with value: 56.0 and parameters: {'0_predictor_instruction': 9, '0_predictor_demos': 5}. Best is trial 5 with value: 72.0.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "Given the context and question, generate a logical answer based on the reasoning process.\n",
            "\n",
            "---\n",
            "\n",
            "Context: The cowboy did not tell other people that he fall off a bronco at the competition.\n",
            "Question: Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This does not necessarily mean that there is a celebrity walking in the city. There could be a non-celebrity person walking in the city.\n",
            "\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This does not necessarily mean that there is a slaver walking in the city. There could be a non-slaver person walking in the city. Additionally, slavery is illegal in most places, so it is highly unlikely that there is a slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m No\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Given the context and question, generate a logical answer based on the reasoning process.\n",
            "\n",
            "---\n",
            "\n",
            "Context: The cowboy did not tell other people that he fall off a bronco at the competition.\n",
            "Question: Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This does not necessarily mean that there is a celebrity walking in the city. There could be a non-celebrity person walking in the city.\n",
            "\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This does not necessarily mean that there is a slaver walking in the city. There could be a non-slaver person walking in the city. Additionally, slavery is illegal in most places, so it is highly unlikely that there is a slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m No\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "...\n",
            "Score 56.0\n",
            "CANDIDATE PROGRAM:\n",
            "Predictor 0\n",
            "i: Please generate a logical answer (either \"Yes\" or \"No\") based on the given context and question using the `ScoNeSignature` function to perform logical reasoning and inference. Take the context and question as input to generate the answer.\n",
            "p: Answer:\n",
            "\n",
            "\n",
            "...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 14 / 25  (56.0): 100%|██████████| 25/25 [00:00<00:00, 1269.37it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FULL TRACE\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 0 / 1  (0.0): 100%|██████████| 1/1 [00:00<00:00, 326.99it/s]\n",
            "[I 2024-06-21 01:06:01,363] Trial 7 finished with value: 56.0 and parameters: {'0_predictor_instruction': 7, '0_predictor_demos': 4}. Best is trial 5 with value: 72.0.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "Please generate a logical answer (either \"Yes\" or \"No\") based on the given context and question using the `ScoNeSignature` function to perform logical reasoning and inference. Take the context and question as input to generate the answer.\n",
            "\n",
            "---\n",
            "\n",
            "Context: The diver has not seen any fish on his dive that did not go too deep.\n",
            "Question: Can we logically conclude for sure that the diver has not seen any tuna on his dive that did not go too deep?\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: The cowboy did not tell other people that he fall off a bronco at the competition.\n",
            "\n",
            "Question: Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the cowboy did not tell other people that he fall off a bronco at the competition. Bronco is a type of horse, so it is a type of horse that the cowboy fell off. Therefore, we can logically conclude that the cowboy did not tell other people that he fall off a horse at the competition.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there is at least one person walking in the city. Slaver is a type of person, so it is a type of person that is walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Please generate a logical answer (either \"Yes\" or \"No\") based on the given context and question using the `ScoNeSignature` function to perform logical reasoning and inference. Take the context and question as input to generate the answer.\n",
            "\n",
            "---\n",
            "\n",
            "Context: The diver has not seen any fish on his dive that did not go too deep.\n",
            "Question: Can we logically conclude for sure that the diver has not seen any tuna on his dive that did not go too deep?\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: The cowboy did not tell other people that he fall off a bronco at the competition.\n",
            "\n",
            "Question: Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the cowboy did not tell other people that he fall off a bronco at the competition. Bronco is a type of horse, so it is a type of horse that the cowboy fell off. Therefore, we can logically conclude that the cowboy did not tell other people that he fall off a horse at the competition.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there is at least one person walking in the city. Slaver is a type of person, so it is a type of person that is walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "...\n",
            "Score 56.0\n",
            "CANDIDATE PROGRAM:\n",
            "Predictor 0\n",
            "i: context, question -> answer\n",
            "p: Answer:\n",
            "\n",
            "\n",
            "...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 18 / 25  (72.0): 100%|██████████| 25/25 [00:00<00:00, 907.74it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FULL TRACE\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 0 / 1  (0.0): 100%|██████████| 1/1 [00:00<00:00, 448.83it/s]\n",
            "[I 2024-06-21 01:06:01,431] Trial 8 finished with value: 72.0 and parameters: {'0_predictor_instruction': 0, '0_predictor_demos': 7}. Best is trial 5 with value: 72.0.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "context, question -> answer\n",
            "\n",
            "---\n",
            "\n",
            "Context: There is not a single person not walking in the city.\n",
            "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
            "\n",
            "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "context, question -> answer\n",
            "\n",
            "---\n",
            "\n",
            "Context: There is not a single person not walking in the city.\n",
            "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
            "\n",
            "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "...\n",
            "Score 72.0\n",
            "CANDIDATE PROGRAM:\n",
            "Predictor 0\n",
            "i: Given the context and question, generate a logical answer based on the reasoning process.\n",
            "p: Answer:\n",
            "\n",
            "\n",
            "...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 20 / 25  (80.0): 100%|██████████| 25/25 [00:00<00:00, 846.48it/s] \n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FULL TRACE\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 0 / 1  (0.0): 100%|██████████| 1/1 [00:00<00:00, 116.36it/s]\n",
            "[I 2024-06-21 01:06:01,509] Trial 9 finished with value: 80.0 and parameters: {'0_predictor_instruction': 9, '0_predictor_demos': 7}. Best is trial 9 with value: 80.0.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "Given the context and question, generate a logical answer based on the reasoning process.\n",
            "\n",
            "---\n",
            "\n",
            "Context: There is not a single person not walking in the city.\n",
            "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
            "\n",
            "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Given the context and question, generate a logical answer based on the reasoning process.\n",
            "\n",
            "---\n",
            "\n",
            "Context: There is not a single person not walking in the city.\n",
            "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
            "\n",
            "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "...\n",
            "Score 80.0\n",
            "CANDIDATE PROGRAM:\n",
            "Predictor 0\n",
            "i: Given the context and question, generate a logical answer based on the reasoning process.\n",
            "p: Answer:\n",
            "\n",
            "\n",
            "...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 19 / 25  (76.0): 100%|██████████| 25/25 [00:00<00:00, 202.29it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FULL TRACE\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 0 / 1  (0.0): 100%|██████████| 1/1 [00:00<00:00, 315.15it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "Given the context and question, generate a logical answer based on the reasoning process.\n",
            "\n",
            "---\n",
            "\n",
            "Context: To believe that a man walks confidently not dropping produce is to believe a falsity.\n",
            "Question: Can we logically conclude for sure that to believe that a man walks confidently not dropping pears is to believe a falsity?\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: A man is not holding anything in his hands.\n",
            "\n",
            "Question: Can we logically conclude for sure that a man is not holding tongues in his hands?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that a man is not holding anything in his hands. This means that he is not holding any physical objects, including tongues. Tongues are not physical objects that can be held in one's hands. Therefore, we can logically conclude that a man is not holding tongues in his hands.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there must be at least one person walking in the city. Since the term \"person\" is a broader category that includes \"slaver\", we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Given the context and question, generate a logical answer based on the reasoning process.\n",
            "\n",
            "---\n",
            "\n",
            "Context: To believe that a man walks confidently not dropping produce is to believe a falsity.\n",
            "Question: Can we logically conclude for sure that to believe that a man walks confidently not dropping pears is to believe a falsity?\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: A man is not holding anything in his hands.\n",
            "\n",
            "Question: Can we logically conclude for sure that a man is not holding tongues in his hands?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that a man is not holding anything in his hands. This means that he is not holding any physical objects, including tongues. Tongues are not physical objects that can be held in one's hands. Therefore, we can logically conclude that a man is not holding tongues in his hands.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there must be at least one person walking in the city. Since the term \"person\" is a broader category that includes \"slaver\", we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "...\n",
            "Score 76.0\n",
            "Best Combination: 2,{0: [[], [Example({'context': 'the boy, not girl, will play an horn, but not for another week', 'question': 'Can we logically conclude for sure that the boy, not girl, will play an instrument, but not for another week?', 'answer': 'Yes', 'category': 'two_not_scoped'}) (input_keys={'context', 'question'}), Example({'context': 'The person did not like raspberries.', 'question': 'Can we logically conclude for sure that the person did not like fruit?', 'answer': 'No', 'category': 'one_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'It is not true that there is not a single person walking in the city.', 'question': 'Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?', 'rationale': 'produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.', 'answer': 'No'}) (input_keys=None), Example({'context': 'It is not true that there is not a single person walking in the city.', 'question': 'Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?', 'answer': 'No', 'category': 'two_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'A man is not standing on top of a ladder that is leaned against a not so tall tree.', 'question': 'Can we logically conclude for sure that a man is not standing on top of a ladder that is leaned against a not so tall pine?', 'rationale': \"produce the answer. We know that the man is not standing on top of a ladder that is leaned against a not so tall tree. We can't conclude that the ladder is leaned against a pine tree, so we can't conclude that the man is not standing on top of a ladder that is leaned against a not so tall pine.\", 'answer': 'No'}) (input_keys=None), Example({'context': 'The people were outside not even trying to keep their voices down, and they woke up a mistress indoors.', 'question': 'Can we logically conclude for sure that the people were outside not even trying to keep their voices down, and they woke up a woman indoors?', 'answer': 'Yes', 'category': 'one_not_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'The cowboy did not tell other people that he fall off a bronco at the competition.', 'question': 'Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?', 'rationale': 'produce the answer. We know that the cowboy did not tell other people that he fall off a bronco at the competition. Bronco is a type of horse, so it is a type of horse that the cowboy fell off. Therefore, we can logically conclude that the cowboy did not tell other people that he fall off a horse at the competition.', 'answer': 'Yes'}) (input_keys=None), Example({'context': 'The diver has not seen any fish on his dive that did not go too deep.', 'question': 'Can we logically conclude for sure that the diver has not seen any tuna on his dive that did not go too deep?', 'answer': 'Yes', 'category': 'one_scoped_one_not_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'It is not true that there is not a single person walking in the city.', 'question': 'Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?', 'rationale': 'produce the answer. We know that it is not true that there is not a single person walking in the city. This does not necessarily mean that there is a celebrity walking in the city. There could be a non-celebrity person walking in the city.', 'answer': 'No'}) (input_keys=None), Example({'context': 'The cowboy did not tell other people that he fall off a bronco at the competition.', 'question': 'Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?', 'answer': 'Yes', 'category': 'one_not_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'There is a boat nearby', 'question': 'Can we logically conclude for sure that there is a kayak nearby?', 'rationale': \"produce the answer. We know that there is a boat nearby, but we don't know what type of boat it is. It could be a kayak, but it could also be a sailboat, a speedboat, or even a rowboat. We can't logically conclude that it is a kayak for sure.\", 'answer': 'No'}) (input_keys=None), Example({'context': 'the boy plays an instrument', 'question': 'Can we logically conclude for sure that the boy plays an flute?', 'answer': 'No', 'category': 'no_negation'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'The men were outside speaking loudly so as to wake the bridesmaid indoors', 'question': 'Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?', 'rationale': 'produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.', 'answer': 'Yes'}) (input_keys=None), Example({'context': 'There is not a single person not walking in the city.', 'question': 'Can we logically conclude for sure that there is not a single matriarch not walking in the city?', 'answer': 'Yes', 'category': 'one_scoped_one_not_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': \"There is a athlete walking in the city when it's not dark.\", 'question': \"Can we logically conclude for sure that there is a person walking in the city when it's not dark?\", 'rationale': 'produce the answer. We know that the athlete is walking in the city, and we know that the athlete is a person. Therefore, we can logically conclude that there is a person walking in the city.', 'answer': 'Yes'}) (input_keys=None), Example({'context': 'The man is not listening to techno.', 'question': 'Can we logically conclude for sure that the man is not listening to music?', 'answer': 'No', 'category': 'one_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'A man is not holding anything in his hands.', 'question': 'Can we logically conclude for sure that a man is not holding tongues in his hands?', 'rationale': \"produce the answer. We know that a man is not holding anything in his hands. This means that he is not holding any physical objects, including tongues. Tongues are not physical objects that can be held in one's hands. Therefore, we can logically conclude that a man is not holding tongues in his hands.\", 'answer': 'Yes'}) (input_keys=None), Example({'context': 'To believe that a man walks confidently not dropping produce is to believe a falsity.', 'question': 'Can we logically conclude for sure that to believe that a man walks confidently not dropping pears is to believe a falsity?', 'answer': 'No', 'category': 'two_scoped'}) (input_keys={'context', 'question'})]]} with Mean = 72.0\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 139 / 200  (69.5): 100%|██████████| 200/200 [00:00<00:00, 691.92it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "UPDATING BEST SCORE WITH 69.5\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 128 / 200  (64.0): 100%|██████████| 200/200 [00:00<00:00, 643.31it/s]\n",
            "[I 2024-06-21 01:06:02,325] Trial 10 finished with value: 76.0 and parameters: {'0_predictor_instruction': 9, '0_predictor_demos': 9}. Best is trial 9 with value: 80.0.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "CANDIDATE PROGRAM:\n",
            "Predictor 0\n",
            "i: Given the context and question, generate a logical answer based on the reasoning process.\n",
            "p: Answer:\n",
            "\n",
            "\n",
            "...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 18 / 25  (72.0): 100%|██████████| 25/25 [00:00<00:00, 1546.18it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FULL TRACE\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 0 / 1  (0.0): 100%|██████████| 1/1 [00:00<00:00, 516.29it/s]\n",
            "[I 2024-06-21 01:06:02,377] Trial 11 finished with value: 72.0 and parameters: {'0_predictor_instruction': 9, '0_predictor_demos': 9}. Best is trial 9 with value: 80.0.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "Given the context and question, generate a logical answer based on the reasoning process.\n",
            "\n",
            "---\n",
            "\n",
            "Context: To believe that a man walks confidently not dropping produce is to believe a falsity.\n",
            "Question: Can we logically conclude for sure that to believe that a man walks confidently not dropping pears is to believe a falsity?\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: A man is not holding anything in his hands.\n",
            "\n",
            "Question: Can we logically conclude for sure that a man is not holding tongues in his hands?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that a man is not holding anything in his hands. This means that he is not holding any physical objects, including tongues. Tongues are not physical objects that can be held in one's hands. Therefore, we can logically conclude that a man is not holding tongues in his hands.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there must be at least one person walking in the city. Since the term \"person\" is a broader category that includes \"slaver\", we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Given the context and question, generate a logical answer based on the reasoning process.\n",
            "\n",
            "---\n",
            "\n",
            "Context: To believe that a man walks confidently not dropping produce is to believe a falsity.\n",
            "Question: Can we logically conclude for sure that to believe that a man walks confidently not dropping pears is to believe a falsity?\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: A man is not holding anything in his hands.\n",
            "\n",
            "Question: Can we logically conclude for sure that a man is not holding tongues in his hands?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that a man is not holding anything in his hands. This means that he is not holding any physical objects, including tongues. Tongues are not physical objects that can be held in one's hands. Therefore, we can logically conclude that a man is not holding tongues in his hands.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there must be at least one person walking in the city. Since the term \"person\" is a broader category that includes \"slaver\", we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "...\n",
            "Score 72.0\n",
            "CANDIDATE PROGRAM:\n",
            "Predictor 0\n",
            "i: Given the context and question, generate a logical answer based on the reasoning process.\n",
            "p: Answer:\n",
            "\n",
            "\n",
            "...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 20 / 25  (80.0): 100%|██████████| 25/25 [00:00<00:00, 1216.49it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FULL TRACE\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 0 / 1  (0.0): 100%|██████████| 1/1 [00:00<00:00, 739.61it/s]\n",
            "[I 2024-06-21 01:06:02,442] Trial 12 finished with value: 80.0 and parameters: {'0_predictor_instruction': 9, '0_predictor_demos': 7}. Best is trial 9 with value: 80.0.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "Given the context and question, generate a logical answer based on the reasoning process.\n",
            "\n",
            "---\n",
            "\n",
            "Context: There is not a single person not walking in the city.\n",
            "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
            "\n",
            "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Given the context and question, generate a logical answer based on the reasoning process.\n",
            "\n",
            "---\n",
            "\n",
            "Context: There is not a single person not walking in the city.\n",
            "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
            "\n",
            "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "...\n",
            "Score 80.0\n",
            "CANDIDATE PROGRAM:\n",
            "Predictor 0\n",
            "i: Given the context and question, generate a logical answer based on the reasoning process.\n",
            "p: Answer:\n",
            "\n",
            "\n",
            "...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 15 / 25  (60.0): 100%|██████████| 25/25 [00:00<00:00, 1431.84it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FULL TRACE\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 0 / 1  (0.0): 100%|██████████| 1/1 [00:00<00:00, 896.79it/s]\n",
            "[I 2024-06-21 01:06:02,490] Trial 13 finished with value: 60.0 and parameters: {'0_predictor_instruction': 9, '0_predictor_demos': 7}. Best is trial 9 with value: 80.0.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "Given the context and question, generate a logical answer based on the reasoning process.\n",
            "\n",
            "---\n",
            "\n",
            "Context: There is not a single person not walking in the city.\n",
            "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
            "\n",
            "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Given the context and question, generate a logical answer based on the reasoning process.\n",
            "\n",
            "---\n",
            "\n",
            "Context: There is not a single person not walking in the city.\n",
            "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
            "\n",
            "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "...\n",
            "Score 60.0\n",
            "CANDIDATE PROGRAM:\n",
            "Predictor 0\n",
            "i: Please use the provided context and question to generate a logical inference for the answer using the language model. The answer should be \"Yes\" or \"No\" based on the logical reasoning and inference from the given context and question.\n",
            "p: Answer:\n",
            "\n",
            "\n",
            "...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 14 / 25  (56.0): 100%|██████████| 25/25 [00:00<00:00, 850.72it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FULL TRACE\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 0 / 1  (0.0): 100%|██████████| 1/1 [00:00<00:00, 398.24it/s]\n",
            "[I 2024-06-21 01:06:02,653] Trial 14 finished with value: 56.0 and parameters: {'0_predictor_instruction': 4, '0_predictor_demos': 7}. Best is trial 9 with value: 80.0.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "Please use the provided context and question to generate a logical inference for the answer using the language model. The answer should be \"Yes\" or \"No\" based on the logical reasoning and inference from the given context and question.\n",
            "\n",
            "---\n",
            "\n",
            "Context: There is not a single person not walking in the city.\n",
            "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
            "\n",
            "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Please use the provided context and question to generate a logical inference for the answer using the language model. The answer should be \"Yes\" or \"No\" based on the logical reasoning and inference from the given context and question.\n",
            "\n",
            "---\n",
            "\n",
            "Context: There is not a single person not walking in the city.\n",
            "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
            "\n",
            "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "...\n",
            "Score 56.0\n",
            "CANDIDATE PROGRAM:\n",
            "Predictor 0\n",
            "i: Given the context and a question, use logical reasoning and inference to determine and provide a yes or no answer based on the information provided.\n",
            "p: Answer:\n",
            "\n",
            "\n",
            "...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 14 / 25  (56.0): 100%|██████████| 25/25 [00:00<00:00, 743.10it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FULL TRACE\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 0 / 1  (0.0): 100%|██████████| 1/1 [00:00<00:00, 117.87it/s]\n",
            "[I 2024-06-21 01:06:02,724] Trial 15 finished with value: 56.0 and parameters: {'0_predictor_instruction': 5, '0_predictor_demos': 0}. Best is trial 9 with value: 80.0.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "Given the context and a question, use logical reasoning and inference to determine and provide a yes or no answer based on the information provided.\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there must be at least one person walking in the city. Now, we are asked if we can logically conclude for sure that it is not true that there is not a single slaver walking in the city. Since we know that there is at least one person walking in the city, we can conclude that it is not true that there is not a single slaver walking in the city. This is because a slaver is a type of person, and we already know that there is at least one person walking in the city.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Given the context and a question, use logical reasoning and inference to determine and provide a yes or no answer based on the information provided.\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there must be at least one person walking in the city. Now, we are asked if we can logically conclude for sure that it is not true that there is not a single slaver walking in the city. Since we know that there is at least one person walking in the city, we can conclude that it is not true that there is not a single slaver walking in the city. This is because a slaver is a type of person, and we already know that there is at least one person walking in the city.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "...\n",
            "Score 56.0\n",
            "CANDIDATE PROGRAM:\n",
            "Predictor 0\n",
            "i: Please generate a logical answer (either \"Yes\" or \"No\") based on the given context and question using the `ScoNeSignature` function to perform logical reasoning and inference. Take the context and question as input to generate the answer.\n",
            "p: Answer:\n",
            "\n",
            "\n",
            "...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 17 / 25  (68.0): 100%|██████████| 25/25 [00:00<00:00, 967.53it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FULL TRACE\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 0 / 1  (0.0): 100%|██████████| 1/1 [00:00<00:00, 608.22it/s]\n",
            "[I 2024-06-21 01:06:02,793] Trial 16 finished with value: 68.0 and parameters: {'0_predictor_instruction': 7, '0_predictor_demos': 7}. Best is trial 9 with value: 80.0.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "Please generate a logical answer (either \"Yes\" or \"No\") based on the given context and question using the `ScoNeSignature` function to perform logical reasoning and inference. Take the context and question as input to generate the answer.\n",
            "\n",
            "---\n",
            "\n",
            "Context: There is not a single person not walking in the city.\n",
            "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
            "\n",
            "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Please generate a logical answer (either \"Yes\" or \"No\") based on the given context and question using the `ScoNeSignature` function to perform logical reasoning and inference. Take the context and question as input to generate the answer.\n",
            "\n",
            "---\n",
            "\n",
            "Context: There is not a single person not walking in the city.\n",
            "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
            "\n",
            "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "...\n",
            "Score 68.0\n",
            "CANDIDATE PROGRAM:\n",
            "Predictor 0\n",
            "i: Given a context and a question, use the provided context and question to determine if the logical conclusion is \"Yes\" or \"No\" and generate the corresponding answer.\n",
            "p: Answer:\n",
            "\n",
            "\n",
            "...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 17 / 25  (68.0): 100%|██████████| 25/25 [00:00<00:00, 1159.33it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FULL TRACE\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 0 / 1  (0.0): 100%|██████████| 1/1 [00:00<00:00, 528.52it/s]\n",
            "[I 2024-06-21 01:06:02,842] Trial 17 finished with value: 68.0 and parameters: {'0_predictor_instruction': 8, '0_predictor_demos': 7}. Best is trial 9 with value: 80.0.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "Given a context and a question, use the provided context and question to determine if the logical conclusion is \"Yes\" or \"No\" and generate the corresponding answer.\n",
            "\n",
            "---\n",
            "\n",
            "Context: There is not a single person not walking in the city.\n",
            "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
            "\n",
            "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Given a context and a question, use the provided context and question to determine if the logical conclusion is \"Yes\" or \"No\" and generate the corresponding answer.\n",
            "\n",
            "---\n",
            "\n",
            "Context: There is not a single person not walking in the city.\n",
            "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
            "\n",
            "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "...\n",
            "Score 68.0\n",
            "CANDIDATE PROGRAM:\n",
            "Predictor 0\n",
            "i: Given the context and question, generate a logical answer based on the reasoning process.\n",
            "p: Answer:\n",
            "\n",
            "\n",
            "...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 18 / 25  (72.0): 100%|██████████| 25/25 [00:00<00:00, 924.62it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FULL TRACE\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 1 / 1  (100.0): 100%|██████████| 1/1 [00:00<00:00, 220.07it/s]\n",
            "[I 2024-06-21 01:06:02,919] Trial 18 finished with value: 72.0 and parameters: {'0_predictor_instruction': 9, '0_predictor_demos': 1}. Best is trial 9 with value: 80.0.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "Given the context and question, generate a logical answer based on the reasoning process.\n",
            "\n",
            "---\n",
            "\n",
            "Context: the boy, not girl, will play an horn, but not for another week\n",
            "Question: Can we logically conclude for sure that the boy, not girl, will play an instrument, but not for another week?\n",
            "Answer: Yes\n",
            "\n",
            "Context: The person did not like raspberries.\n",
            "Question: Can we logically conclude for sure that the person did not like fruit?\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. This means that there is at least one person walking in the city. However, we cannot logically conclude that there is not a single slaver walking in the city. We only know that there is at least one person, but we don't have any information about the presence of slavers.\n",
            "\n",
            "Answer:\u001b[32m No\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Given the context and question, generate a logical answer based on the reasoning process.\n",
            "\n",
            "---\n",
            "\n",
            "Context: the boy, not girl, will play an horn, but not for another week\n",
            "Question: Can we logically conclude for sure that the boy, not girl, will play an instrument, but not for another week?\n",
            "Answer: Yes\n",
            "\n",
            "Context: The person did not like raspberries.\n",
            "Question: Can we logically conclude for sure that the person did not like fruit?\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. This means that there is at least one person walking in the city. However, we cannot logically conclude that there is not a single slaver walking in the city. We only know that there is at least one person, but we don't have any information about the presence of slavers.\n",
            "\n",
            "Answer:\u001b[32m No\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "...\n",
            "Score 72.0\n",
            "CANDIDATE PROGRAM:\n",
            "Predictor 0\n",
            "i: Given the context and question, generate a logical answer based on the reasoning process.\n",
            "p: Answer:\n",
            "\n",
            "\n",
            "...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 17 / 25  (68.0): 100%|██████████| 25/25 [00:00<00:00, 2253.45it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FULL TRACE\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 1 / 1  (100.0): 100%|██████████| 1/1 [00:00<00:00, 285.21it/s]\n",
            "[I 2024-06-21 01:06:03,017] Trial 19 finished with value: 68.0 and parameters: {'0_predictor_instruction': 9, '0_predictor_demos': 8}. Best is trial 9 with value: 80.0.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "Given the context and question, generate a logical answer based on the reasoning process.\n",
            "\n",
            "---\n",
            "\n",
            "Context: The man is not listening to techno.\n",
            "Question: Can we logically conclude for sure that the man is not listening to music?\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: There is a athlete walking in the city when it's not dark.\n",
            "\n",
            "Question: Can we logically conclude for sure that there is a person walking in the city when it's not dark?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the athlete is walking in the city, and we know that the athlete is a person. Therefore, we can logically conclude that there is a person walking in the city.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. However, we cannot logically conclude that it is not true that there is not a single slaver walking in the city, because we have no information about the presence of slavers in the city.\n",
            "\n",
            "Answer:\u001b[32m No\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Given the context and question, generate a logical answer based on the reasoning process.\n",
            "\n",
            "---\n",
            "\n",
            "Context: The man is not listening to techno.\n",
            "Question: Can we logically conclude for sure that the man is not listening to music?\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: There is a athlete walking in the city when it's not dark.\n",
            "\n",
            "Question: Can we logically conclude for sure that there is a person walking in the city when it's not dark?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the athlete is walking in the city, and we know that the athlete is a person. Therefore, we can logically conclude that there is a person walking in the city.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. However, we cannot logically conclude that it is not true that there is not a single slaver walking in the city, because we have no information about the presence of slavers in the city.\n",
            "\n",
            "Answer:\u001b[32m No\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "...\n",
            "Score 68.0\n",
            "CANDIDATE PROGRAM:\n",
            "Predictor 0\n",
            "i: Determine if it can be logically concluded that a kayak is nearby based on the given context and question, and generate a yes or no answer using the language model.\n",
            "p: Answer:\n",
            "\n",
            "\n",
            "...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 18 / 25  (72.0): 100%|██████████| 25/25 [00:00<00:00, 1312.95it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FULL TRACE\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 0 / 1  (0.0): 100%|██████████| 1/1 [00:00<00:00, 624.25it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "Determine if it can be logically concluded that a kayak is nearby based on the given context and question, and generate a yes or no answer using the language model.\n",
            "\n",
            "---\n",
            "\n",
            "Context: There is not a single person not walking in the city.\n",
            "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
            "\n",
            "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Determine if it can be logically concluded that a kayak is nearby based on the given context and question, and generate a yes or no answer using the language model.\n",
            "\n",
            "---\n",
            "\n",
            "Context: There is not a single person not walking in the city.\n",
            "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
            "\n",
            "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "...\n",
            "Score 72.0\n",
            "Best Combination: 0,{0: [[], [Example({'context': 'the boy, not girl, will play an horn, but not for another week', 'question': 'Can we logically conclude for sure that the boy, not girl, will play an instrument, but not for another week?', 'answer': 'Yes', 'category': 'two_not_scoped'}) (input_keys={'context', 'question'}), Example({'context': 'The person did not like raspberries.', 'question': 'Can we logically conclude for sure that the person did not like fruit?', 'answer': 'No', 'category': 'one_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'It is not true that there is not a single person walking in the city.', 'question': 'Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?', 'rationale': 'produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.', 'answer': 'No'}) (input_keys=None), Example({'context': 'It is not true that there is not a single person walking in the city.', 'question': 'Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?', 'answer': 'No', 'category': 'two_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'A man is not standing on top of a ladder that is leaned against a not so tall tree.', 'question': 'Can we logically conclude for sure that a man is not standing on top of a ladder that is leaned against a not so tall pine?', 'rationale': \"produce the answer. We know that the man is not standing on top of a ladder that is leaned against a not so tall tree. We can't conclude that the ladder is leaned against a pine tree, so we can't conclude that the man is not standing on top of a ladder that is leaned against a not so tall pine.\", 'answer': 'No'}) (input_keys=None), Example({'context': 'The people were outside not even trying to keep their voices down, and they woke up a mistress indoors.', 'question': 'Can we logically conclude for sure that the people were outside not even trying to keep their voices down, and they woke up a woman indoors?', 'answer': 'Yes', 'category': 'one_not_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'The cowboy did not tell other people that he fall off a bronco at the competition.', 'question': 'Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?', 'rationale': 'produce the answer. We know that the cowboy did not tell other people that he fall off a bronco at the competition. Bronco is a type of horse, so it is a type of horse that the cowboy fell off. Therefore, we can logically conclude that the cowboy did not tell other people that he fall off a horse at the competition.', 'answer': 'Yes'}) (input_keys=None), Example({'context': 'The diver has not seen any fish on his dive that did not go too deep.', 'question': 'Can we logically conclude for sure that the diver has not seen any tuna on his dive that did not go too deep?', 'answer': 'Yes', 'category': 'one_scoped_one_not_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'It is not true that there is not a single person walking in the city.', 'question': 'Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?', 'rationale': 'produce the answer. We know that it is not true that there is not a single person walking in the city. This does not necessarily mean that there is a celebrity walking in the city. There could be a non-celebrity person walking in the city.', 'answer': 'No'}) (input_keys=None), Example({'context': 'The cowboy did not tell other people that he fall off a bronco at the competition.', 'question': 'Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?', 'answer': 'Yes', 'category': 'one_not_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'There is a boat nearby', 'question': 'Can we logically conclude for sure that there is a kayak nearby?', 'rationale': \"produce the answer. We know that there is a boat nearby, but we don't know what type of boat it is. It could be a kayak, but it could also be a sailboat, a speedboat, or even a rowboat. We can't logically conclude that it is a kayak for sure.\", 'answer': 'No'}) (input_keys=None), Example({'context': 'the boy plays an instrument', 'question': 'Can we logically conclude for sure that the boy plays an flute?', 'answer': 'No', 'category': 'no_negation'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'The men were outside speaking loudly so as to wake the bridesmaid indoors', 'question': 'Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?', 'rationale': 'produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.', 'answer': 'Yes'}) (input_keys=None), Example({'context': 'There is not a single person not walking in the city.', 'question': 'Can we logically conclude for sure that there is not a single matriarch not walking in the city?', 'answer': 'Yes', 'category': 'one_scoped_one_not_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': \"There is a athlete walking in the city when it's not dark.\", 'question': \"Can we logically conclude for sure that there is a person walking in the city when it's not dark?\", 'rationale': 'produce the answer. We know that the athlete is walking in the city, and we know that the athlete is a person. Therefore, we can logically conclude that there is a person walking in the city.', 'answer': 'Yes'}) (input_keys=None), Example({'context': 'The man is not listening to techno.', 'question': 'Can we logically conclude for sure that the man is not listening to music?', 'answer': 'No', 'category': 'one_scoped'}) (input_keys={'context', 'question'})], [Example({'augmented': True, 'context': 'A man is not holding anything in his hands.', 'question': 'Can we logically conclude for sure that a man is not holding tongues in his hands?', 'rationale': \"produce the answer. We know that a man is not holding anything in his hands. This means that he is not holding any physical objects, including tongues. Tongues are not physical objects that can be held in one's hands. Therefore, we can logically conclude that a man is not holding tongues in his hands.\", 'answer': 'Yes'}) (input_keys=None), Example({'context': 'To believe that a man walks confidently not dropping produce is to believe a falsity.', 'question': 'Can we logically conclude for sure that to believe that a man walks confidently not dropping pears is to believe a falsity?', 'answer': 'No', 'category': 'two_scoped'}) (input_keys={'context', 'question'})]]} with Mean = 72.0\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 142 / 200  (71.0): 100%|██████████| 200/200 [00:00<00:00, 725.50it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "UPDATING BEST SCORE WITH 71.0\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 130 / 200  (65.0): 100%|██████████| 200/200 [00:00<00:00, 738.21it/s]\n",
            "[I 2024-06-21 01:06:03,753] Trial 20 finished with value: 72.0 and parameters: {'0_predictor_instruction': 6, '0_predictor_demos': 7}. Best is trial 9 with value: 80.0.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "CANDIDATE PROGRAM:\n",
            "Predictor 0\n",
            "i: Given the context and a question, use logical reasoning and inference to determine and provide a yes or no answer based on the information provided.\n",
            "p: Answer:\n",
            "\n",
            "\n",
            "...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 19 / 25  (76.0): 100%|██████████| 25/25 [00:00<00:00, 677.20it/s] \n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FULL TRACE\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 0 / 1  (0.0): 100%|██████████| 1/1 [00:00<00:00, 525.14it/s]\n",
            "[I 2024-06-21 01:06:03,818] Trial 21 finished with value: 76.0 and parameters: {'0_predictor_instruction': 5, '0_predictor_demos': 9}. Best is trial 9 with value: 80.0.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "Given the context and a question, use logical reasoning and inference to determine and provide a yes or no answer based on the information provided.\n",
            "\n",
            "---\n",
            "\n",
            "Context: To believe that a man walks confidently not dropping produce is to believe a falsity.\n",
            "Question: Can we logically conclude for sure that to believe that a man walks confidently not dropping pears is to believe a falsity?\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: A man is not holding anything in his hands.\n",
            "\n",
            "Question: Can we logically conclude for sure that a man is not holding tongues in his hands?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that a man is not holding anything in his hands. This means that he is not holding any physical objects, including tongues. Tongues are not physical objects that can be held in one's hands. Therefore, we can logically conclude that a man is not holding tongues in his hands.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there must be at least one person walking in the city. Since the term \"person\" is a broader category that includes \"slaver\", we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Given the context and a question, use logical reasoning and inference to determine and provide a yes or no answer based on the information provided.\n",
            "\n",
            "---\n",
            "\n",
            "Context: To believe that a man walks confidently not dropping produce is to believe a falsity.\n",
            "Question: Can we logically conclude for sure that to believe that a man walks confidently not dropping pears is to believe a falsity?\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: A man is not holding anything in his hands.\n",
            "\n",
            "Question: Can we logically conclude for sure that a man is not holding tongues in his hands?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that a man is not holding anything in his hands. This means that he is not holding any physical objects, including tongues. Tongues are not physical objects that can be held in one's hands. Therefore, we can logically conclude that a man is not holding tongues in his hands.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there must be at least one person walking in the city. Since the term \"person\" is a broader category that includes \"slaver\", we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "...\n",
            "Score 76.0\n",
            "CANDIDATE PROGRAM:\n",
            "Predictor 0\n",
            "i: Given the context and question, generate a logical answer based on the reasoning process.\n",
            "p: Answer:\n",
            "\n",
            "\n",
            "...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 19 / 25  (76.0): 100%|██████████| 25/25 [00:00<00:00, 887.00it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FULL TRACE\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 0 / 1  (0.0): 100%|██████████| 1/1 [00:00<00:00, 590.00it/s]\n",
            "[I 2024-06-21 01:06:03,871] Trial 22 finished with value: 76.0 and parameters: {'0_predictor_instruction': 9, '0_predictor_demos': 4}. Best is trial 9 with value: 80.0.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "Given the context and question, generate a logical answer based on the reasoning process.\n",
            "\n",
            "---\n",
            "\n",
            "Context: The diver has not seen any fish on his dive that did not go too deep.\n",
            "Question: Can we logically conclude for sure that the diver has not seen any tuna on his dive that did not go too deep?\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: The cowboy did not tell other people that he fall off a bronco at the competition.\n",
            "\n",
            "Question: Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the cowboy did not tell other people that he fall off a bronco at the competition. Bronco is a type of horse, so it is a type of horse that the cowboy fell off. Therefore, we can logically conclude that the cowboy did not tell other people that he fall off a horse at the competition.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there is at least one person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city, because there is at least one person walking in the city, and that person could be a slaver.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Given the context and question, generate a logical answer based on the reasoning process.\n",
            "\n",
            "---\n",
            "\n",
            "Context: The diver has not seen any fish on his dive that did not go too deep.\n",
            "Question: Can we logically conclude for sure that the diver has not seen any tuna on his dive that did not go too deep?\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: The cowboy did not tell other people that he fall off a bronco at the competition.\n",
            "\n",
            "Question: Can we logically conclude for sure that the cowboy did not tell other people that he fall off a horse at the competition?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the cowboy did not tell other people that he fall off a bronco at the competition. Bronco is a type of horse, so it is a type of horse that the cowboy fell off. Therefore, we can logically conclude that the cowboy did not tell other people that he fall off a horse at the competition.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there is at least one person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city, because there is at least one person walking in the city, and that person could be a slaver.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "...\n",
            "Score 76.0\n",
            "CANDIDATE PROGRAM:\n",
            "Predictor 0\n",
            "i: context, question -> answer\n",
            "p: Answer:\n",
            "\n",
            "\n",
            "...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 22 / 25  (88.0): 100%|██████████| 25/25 [00:00<00:00, 1023.58it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FULL TRACE\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 1 / 1  (100.0): 100%|██████████| 1/1 [00:00<00:00, 156.45it/s]\n",
            "[I 2024-06-21 01:06:03,944] Trial 23 finished with value: 88.0 and parameters: {'0_predictor_instruction': 0, '0_predictor_demos': 9}. Best is trial 23 with value: 88.0.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "context, question -> answer\n",
            "\n",
            "---\n",
            "\n",
            "Context: To believe that a man walks confidently not dropping produce is to believe a falsity.\n",
            "Question: Can we logically conclude for sure that to believe that a man walks confidently not dropping pears is to believe a falsity?\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: A man is not holding anything in his hands.\n",
            "\n",
            "Question: Can we logically conclude for sure that a man is not holding tongues in his hands?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that a man is not holding anything in his hands. This means that he is not holding any physical objects, including tongues. Tongues are not physical objects that can be held in one's hands. Therefore, we can logically conclude that a man is not holding tongues in his hands.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there is at least one person walking in the city. We cannot logically conclude that there is at least one slaver walking in the city, as the original statement does not mention anything about slavery. Therefore, we cannot logically conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m No\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "context, question -> answer\n",
            "\n",
            "---\n",
            "\n",
            "Context: To believe that a man walks confidently not dropping produce is to believe a falsity.\n",
            "Question: Can we logically conclude for sure that to believe that a man walks confidently not dropping pears is to believe a falsity?\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: A man is not holding anything in his hands.\n",
            "\n",
            "Question: Can we logically conclude for sure that a man is not holding tongues in his hands?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that a man is not holding anything in his hands. This means that he is not holding any physical objects, including tongues. Tongues are not physical objects that can be held in one's hands. Therefore, we can logically conclude that a man is not holding tongues in his hands.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there is at least one person walking in the city. We cannot logically conclude that there is at least one slaver walking in the city, as the original statement does not mention anything about slavery. Therefore, we cannot logically conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m No\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "...\n",
            "Score 88.0\n",
            "CANDIDATE PROGRAM:\n",
            "Predictor 0\n",
            "i: context, question -> answer\n",
            "p: Answer:\n",
            "\n",
            "\n",
            "...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 18 / 25  (72.0): 100%|██████████| 25/25 [00:00<00:00, 1211.71it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FULL TRACE\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 1 / 1  (100.0): 100%|██████████| 1/1 [00:00<00:00, 429.13it/s]\n",
            "[I 2024-06-21 01:06:04,004] Trial 24 finished with value: 72.0 and parameters: {'0_predictor_instruction': 0, '0_predictor_demos': 9}. Best is trial 23 with value: 88.0.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "context, question -> answer\n",
            "\n",
            "---\n",
            "\n",
            "Context: To believe that a man walks confidently not dropping produce is to believe a falsity.\n",
            "Question: Can we logically conclude for sure that to believe that a man walks confidently not dropping pears is to believe a falsity?\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: A man is not holding anything in his hands.\n",
            "\n",
            "Question: Can we logically conclude for sure that a man is not holding tongues in his hands?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that a man is not holding anything in his hands. This means that he is not holding any physical objects, including tongues. Tongues are not physical objects that can be held in one's hands. Therefore, we can logically conclude that a man is not holding tongues in his hands.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there is at least one person walking in the city. We cannot logically conclude that there is at least one slaver walking in the city, as the original statement does not mention anything about slavery. Therefore, we cannot logically conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m No\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "context, question -> answer\n",
            "\n",
            "---\n",
            "\n",
            "Context: To believe that a man walks confidently not dropping produce is to believe a falsity.\n",
            "Question: Can we logically conclude for sure that to believe that a man walks confidently not dropping pears is to believe a falsity?\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: A man is not holding anything in his hands.\n",
            "\n",
            "Question: Can we logically conclude for sure that a man is not holding tongues in his hands?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that a man is not holding anything in his hands. This means that he is not holding any physical objects, including tongues. Tongues are not physical objects that can be held in one's hands. Therefore, we can logically conclude that a man is not holding tongues in his hands.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there is at least one person walking in the city. We cannot logically conclude that there is at least one slaver walking in the city, as the original statement does not mention anything about slavery. Therefore, we cannot logically conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m No\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "...\n",
            "Score 72.0\n",
            "CANDIDATE PROGRAM:\n",
            "Predictor 0\n",
            "i: Given a context and a question, use logical reasoning to determine if a logical conclusion can be made and generate a \"yes\" or \"no\" answer based on the provided information.\n",
            "p: Answer:\n",
            "\n",
            "\n",
            "...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 18 / 25  (72.0): 100%|██████████| 25/25 [00:00<00:00, 1147.59it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FULL TRACE\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 0 / 1  (0.0): 100%|██████████| 1/1 [00:00<00:00, 248.52it/s]\n",
            "[I 2024-06-21 01:06:04,177] Trial 25 finished with value: 72.0 and parameters: {'0_predictor_instruction': 3, '0_predictor_demos': 7}. Best is trial 23 with value: 88.0.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "Given a context and a question, use logical reasoning to determine if a logical conclusion can be made and generate a \"yes\" or \"no\" answer based on the provided information.\n",
            "\n",
            "---\n",
            "\n",
            "Context: There is not a single person not walking in the city.\n",
            "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
            "\n",
            "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Given a context and a question, use logical reasoning to determine if a logical conclusion can be made and generate a \"yes\" or \"no\" answer based on the provided information.\n",
            "\n",
            "---\n",
            "\n",
            "Context: There is not a single person not walking in the city.\n",
            "Question: Can we logically conclude for sure that there is not a single matriarch not walking in the city?\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: The men were outside speaking loudly so as to wake the bridesmaid indoors\n",
            "\n",
            "Question: Can we logically conclude for sure that the men were outside speaking loudly so as to wake the woman indoors?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the men were outside speaking loudly, and we also know that the bridesmaid is a woman. Therefore, we can logically conclude that the men were outside speaking loudly so as to wake the woman indoors.\n",
            "\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. Therefore, we can logically conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "...\n",
            "Score 72.0\n",
            "CANDIDATE PROGRAM:\n",
            "Predictor 0\n",
            "i: context, question -> answer\n",
            "p: Answer:\n",
            "\n",
            "\n",
            "...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 17 / 25  (68.0): 100%|██████████| 25/25 [00:00<00:00, 1022.33it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FULL TRACE\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 1 / 1  (100.0): 100%|██████████| 1/1 [00:00<00:00, 216.73it/s]\n",
            "[I 2024-06-21 01:06:04,245] Trial 26 finished with value: 68.0 and parameters: {'0_predictor_instruction': 0, '0_predictor_demos': 3}. Best is trial 23 with value: 88.0.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "context, question -> answer\n",
            "\n",
            "---\n",
            "\n",
            "Context: The people were outside not even trying to keep their voices down, and they woke up a mistress indoors.\n",
            "Question: Can we logically conclude for sure that the people were outside not even trying to keep their voices down, and they woke up a woman indoors?\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: A man is not standing on top of a ladder that is leaned against a not so tall tree.\n",
            "\n",
            "Question: Can we logically conclude for sure that a man is not standing on top of a ladder that is leaned against a not so tall pine?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the man is not standing on top of a ladder that is leaned against a not so tall tree. We can't conclude that the ladder is leaned against a pine tree, so we can't conclude that the man is not standing on top of a ladder that is leaned against a not so tall pine.\n",
            "\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. We can't conclude that the person is a slaver, so we can't conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m No\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "context, question -> answer\n",
            "\n",
            "---\n",
            "\n",
            "Context: The people were outside not even trying to keep their voices down, and they woke up a mistress indoors.\n",
            "Question: Can we logically conclude for sure that the people were outside not even trying to keep their voices down, and they woke up a woman indoors?\n",
            "Answer: Yes\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: A man is not standing on top of a ladder that is leaned against a not so tall tree.\n",
            "\n",
            "Question: Can we logically conclude for sure that a man is not standing on top of a ladder that is leaned against a not so tall pine?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the man is not standing on top of a ladder that is leaned against a not so tall tree. We can't conclude that the ladder is leaned against a pine tree, so we can't conclude that the man is not standing on top of a ladder that is leaned against a not so tall pine.\n",
            "\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. We can't conclude that the person is a slaver, so we can't conclude that it is not true that there is not a single slaver walking in the city.\n",
            "\n",
            "Answer:\u001b[32m No\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "...\n",
            "Score 68.0\n",
            "CANDIDATE PROGRAM:\n",
            "Predictor 0\n",
            "i: context, question -> answer\n",
            "p: Answer:\n",
            "\n",
            "\n",
            "...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 9 / 25  (36.0): 100%|██████████| 25/25 [00:00<00:00, 1336.67it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FULL TRACE\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 1 / 1  (100.0): 100%|██████████| 1/1 [00:00<00:00, 53.79it/s]\n",
            "[I 2024-06-21 01:06:04,322] Trial 27 finished with value: 36.0 and parameters: {'0_predictor_instruction': 0, '0_predictor_demos': 6}. Best is trial 23 with value: 88.0.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "context, question -> answer\n",
            "\n",
            "---\n",
            "\n",
            "Context: the boy plays an instrument\n",
            "Question: Can we logically conclude for sure that the boy plays an flute?\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: There is a boat nearby\n",
            "\n",
            "Question: Can we logically conclude for sure that there is a kayak nearby?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that there is a boat nearby, but we don't know what type of boat it is. It could be a kayak, but it could also be a sailboat, a speedboat, or even a rowboat. We can't logically conclude that it is a kayak for sure.\n",
            "\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. However, we can't logically conclude that it is not true that there is not a single slaver walking in the city. We don't have any information about the presence of slavers in the city.\n",
            "\n",
            "Answer:\u001b[32m No\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "context, question -> answer\n",
            "\n",
            "---\n",
            "\n",
            "Context: the boy plays an instrument\n",
            "Question: Can we logically conclude for sure that the boy plays an flute?\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: There is a boat nearby\n",
            "\n",
            "Question: Can we logically conclude for sure that there is a kayak nearby?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that there is a boat nearby, but we don't know what type of boat it is. It could be a kayak, but it could also be a sailboat, a speedboat, or even a rowboat. We can't logically conclude that it is a kayak for sure.\n",
            "\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. However, we can't logically conclude that it is not true that there is not a single slaver walking in the city. We don't have any information about the presence of slavers in the city.\n",
            "\n",
            "Answer:\u001b[32m No\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "...\n",
            "Score 36.0\n",
            "CANDIDATE PROGRAM:\n",
            "Predictor 0\n",
            "i: Provide a yes or no answer based on the given context and question. The context and question are used as input to generate the answer.\n",
            "p: Answer:\n",
            "\n",
            "\n",
            "...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 16 / 25  (64.0): 100%|██████████| 25/25 [00:00<00:00, 1755.88it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FULL TRACE\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 0 / 1  (0.0): 100%|██████████| 1/1 [00:00<00:00, 164.77it/s]\n",
            "[I 2024-06-21 01:06:04,385] Trial 28 finished with value: 64.0 and parameters: {'0_predictor_instruction': 1, '0_predictor_demos': 0}. Best is trial 23 with value: 88.0.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "Provide a yes or no answer based on the given context and question. The context and question are used as input to generate the answer.\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there must be at least one person walking in the city. Now, we are asked if we can logically conclude for sure that it is not true that there is not a single slaver walking in the city. Since we know that there is at least one person walking in the city, we can conclude that it is not true that there is not a single slaver walking in the city. This is because a slaver is a type of person, and we know that there is at least one person walking in the city.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "Provide a yes or no answer based on the given context and question. The context and question are used as input to generate the answer.\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that it is not true that there is not a single person walking in the city. This means that there must be at least one person walking in the city. Now, we are asked if we can logically conclude for sure that it is not true that there is not a single slaver walking in the city. Since we know that there is at least one person walking in the city, we can conclude that it is not true that there is not a single slaver walking in the city. This is because a slaver is a type of person, and we know that there is at least one person walking in the city.\n",
            "\n",
            "Answer:\u001b[32m Yes\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "...\n",
            "Score 64.0\n",
            "CANDIDATE PROGRAM:\n",
            "Predictor 0\n",
            "i: context, question -> answer\n",
            "p: Answer:\n",
            "\n",
            "\n",
            "...\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 11 / 25  (44.0): 100%|██████████| 25/25 [00:00<00:00, 1164.42it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FULL TRACE\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 1 / 1  (100.0): 100%|██████████| 1/1 [00:00<00:00, 499.80it/s]\n",
            "[I 2024-06-21 01:06:04,442] Trial 29 finished with value: 44.0 and parameters: {'0_predictor_instruction': 0, '0_predictor_demos': 2}. Best is trial 23 with value: 88.0.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\n",
            "context, question -> answer\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.\n",
            "\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.\n",
            "\n",
            "Answer:\u001b[32m No\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "\n",
            "context, question -> answer\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single celebrity walking in the city?\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Follow the following format.\n",
            "\n",
            "Context: ${context}\n",
            "\n",
            "Question: ${question}\n",
            "\n",
            "Reasoning: Let's think step by step in order to ${produce the answer}. We ...\n",
            "\n",
            "Answer: Yes or No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.\n",
            "\n",
            "Answer: No\n",
            "\n",
            "---\n",
            "\n",
            "Context: It is not true that there is not a single person walking in the city.\n",
            "\n",
            "Question: Can we logically conclude for sure that it is not true that there is not a single slaver walking in the city?\n",
            "\n",
            "Reasoning: Let's think step by step in order to produce the answer. We know that the statement \"there is not a single person walking in the city\" is false. However, we cannot logically conclude that the statement \"there is not a single slaver walking in the city\" is also false. There could be a single person walking in the city who is not a slaver.\n",
            "\n",
            "Answer:\u001b[32m No\u001b[0m\n",
            "\n",
            "\n",
            "\n",
            "...\n",
            "Score 44.0\n"
          ]
        }
      ],
      "source": [
        "import cloudpickle as pickle\n",
        "from dspy.teleprompt import MIPROv2\n",
        "\n",
        "LOAD_PRECOMPILED_PROGRAM = False\n",
        "compiled_program = program.deepcopy()\n",
        "\n",
        "# We can load the precompiled program, but since scone is quick to compile, we can also compile it from scratch\n",
        "if LOAD_PRECOMPILED_PROGRAM:\n",
        "    # Load the data from the file\n",
        "    compiled_program.load(compiled_program_file_path)\n",
        "    with open(trial_logs_file_path, \"rb\") as f:\n",
        "        trial_logs = pickle.load(f)\n",
        "    compiled_program.trial_logs = trial_logs\n",
        "# Otherwise, if desired, the program can be compiled from scratch\n",
        "else:\n",
        "    # Define hyperparameters:\n",
        "    N = 10 # The number of instructions and fewshot examples that we will generate and optimize over\n",
        "    batches = 30 # The number of optimization batches to be run (we will test out a new combination of instructions and fewshot examples in each trial)\n",
        "    temperature = 1.0 # The temperature configured for generating new instructions\n",
        "\n",
        "    # Compile\n",
        "    eval_kwargs = dict(num_threads=16, display_progress=True, display_table=0)\n",
        "    teleprompter = MIPROv2(prompt_model=prompt_model, task_model=task_model, metric=metric, num_candidates=N, init_temperature=temperature, verbose=True)\n",
        "    print(trainset[:10])\n",
        "    compiled_program = teleprompter.compile(program, trainset=trainset, valset=valset, num_batches=batches, max_bootstrapped_demos=1,max_labeled_demos=2, eval_kwargs=eval_kwargs)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uqVVnaEBqZH3"
      },
      "source": [
        "#### 3b] Evaluate optimized program\n",
        "Now, we evaluate our program that has been optimized with MIPRO. We see that performance on train and dev have improved by __+13.0pt__, __+15.5pt__, and __+15.5pt__ respectively!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "VvnBp7huqZH3"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Average Metric: 142 / 200  (71.0): 100%|██████████| 200/200 [00:00<00:00, 792.69it/s]\n",
            "Average Metric: 130 / 200  (65.0): 100%|██████████| 200/200 [00:00<00:00, 665.92it/s]\n",
            "Average Metric: 141 / 200  (70.5): 100%|██████████| 200/200 [00:00<00:00, 818.67it/s]\n"
          ]
        }
      ],
      "source": [
        "bayesian_train_score = evaluate(compiled_program, devset=trainset)\n",
        "bayesian_val_score = evaluate(compiled_program, devset=valset)\n",
        "bayesian_test_score = evaluate(compiled_program, devset=testset)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j3UWn_UnqZH4"
      },
      "source": [
        "#### 3c] Visualizing scores & prompts over trials"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pBYTLTwWqZH4"
      },
      "source": [
        "Now, let's take a look at how this optimization looked over the course of each trial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "rtMUNeicqZH4"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAckAAAE8CAYAAACrYErbAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAABFeklEQVR4nO3deViU5f4/8PeAMKwDiiggq+JC5pK4hAqKouCWCmYuJeVWqSXSJnUMNculMsnMTstBTx5cQ07mFlEquKUYLicz8ovigmgWDEvAyNy/P/wxOcIgMww8A/N+XRfX5dzPM/d8Ps/j8OHZ7lsmhBAgIiKiaiykDoCIiMhUsUgSERHpwCJJRESkA4skERGRDiySREREOrBIEhER6cAiSUREpAOLJBERkQ4skkRERDqwSBI1ERs2bIBMJsPJkyelDoXIbLBIEtWgqiDd+9OmTRuEhoZi7969Bvf7zjvvICUlxXiB6ikjIwMjRoxAu3btYGNjA29vb4wZMwZJSUmSxURkylgkiWqxdOlSfPnll/j3v/+NV199Fbdu3cLIkSPxzTffGNSflEVy+/btCAkJQX5+PubPn4+1a9fiySefxJ9//onPPvtMkpiITF0LqQMgMmUjRoxA7969Na9nzJiBtm3bYvPmzRg9erSEkelv8eLFeOihh3Ds2DFYW1trLbt582ajxSGEQFlZGWxtbRvtM4kMxSNJIj04OzvD1tYWLVpo/3353nvvoX///nBxcYGtrS0CAwOxY8cOrXVkMhlKSkqwceNGzSncp59+WrP82rVrmDFjBjw8PCCXy+Hn54fnn38eFRUVWv2Ul5cjNjYWrq6usLe3x/jx43Hr1q0Hxn7x4kX06dOnWoEEgDZt2mi9VqvVSEhIQLdu3WBjYwNXV1dERERoXQ+9c+cO3nrrLXTo0AFyuRy+vr54/fXXUV5ertWXr68vRo8ejf3796N3796wtbXFP//5TwBAQUEBYmJi4OXlBblcDn9/f6xcuRJqtVqrjy1btiAwMBCOjo5QKBTo1q0bEhISHpgzUX3xSJKoFoWFhfj9998hhMDNmzexdu1aFBcX48knn9RaLyEhAY899himTp2KiooKbNmyBY8//ji++eYbjBo1CgDw5ZdfYubMmejbty9mz54NAOjQoQMA4Pr16+jbty8KCgowe/ZsdOnSBdeuXcOOHTtQWlqqVdheeOEFtGzZEvHx8bh06RLWrFmDefPmYevWrbXm4uPjg7S0NFy9ehWenp61rjtjxgxs2LABI0aMwMyZM3Hnzh2kp6fj2LFjmiPrmTNnYuPGjZgwYQJeeuklHD9+HMuXL8f58+exc+dOrf4uXLiAyZMn49lnn8WsWbPQuXNnlJaWYtCgQbh27RqeffZZeHt748iRI4iLi0NeXh7WrFkDAEhNTcXkyZMxdOhQrFy5EgBw/vx5HD58GPPnz681D6J6E0RUTWJiogBQ7Ucul4sNGzZUW7+0tFTrdUVFhXj44YfFkCFDtNrt7e1FdHR0tfdPmzZNWFhYiBMnTlRbplartWIKCwvTtAkhxIIFC4SlpaUoKCioNacvvvhCABDW1tYiNDRULFq0SKSnp4vKykqt9b7//nsBQLz44os6Y8nKyhIAxMyZM7WWv/zyywKA+P777zVtPj4+AoDYt2+f1rpvvfWWsLe3F7/++qtW+8KFC4WlpaXIzc0VQggxf/58oVAoxJ07d2rNj6gh8HQrUS3WrVuH1NRUpKamYtOmTQgNDcXMmTORnJystd6919f+/PNPFBYWIjg4GKdOnXrgZ6jVaqSkpGDMmDFa1z+ryGQyrdezZ8/WagsODkZlZSUuX75c6+dMnz4d+/btw+DBg5GRkYG33noLwcHB6NixI44cOaJZ76uvvoJMJkN8fLzOWPbs2QMAiI2N1Vr+0ksvAQB2796t1e7n54fw8HCttu3btyM4OBgtW7bE77//rvkJCwtDZWUlDh06BODuKe6SkhKkpqbWmh9RQ+DpVqJa9O3bV6twTZ48GY888gjmzZuH0aNHa06DfvPNN1i2bBmysrK0rsndX+BqcuvWLSiVSjz88MN1isnb21vrdcuWLQHcLc4PEh4ejvDwcJSWliIzMxNbt27FJ598gtGjR+OXX35BmzZtcPHiRXh4eKBVq1Y6+7l8+TIsLCzg7++v1e7m5gZnZ+dqBdvPz69aH9nZ2Thz5gxcXV1r/Iyqm4nmzJmDbdu2aR5dGT58OCZOnIiIiIgH5ktUXyySRHqwsLBAaGgoEhISkJ2dja5duyI9PR2PPfYYQkJC8PHHH8Pd3R1WVlZITExskOcPLS0ta2wXQtS5Dzs7OwQHByM4OBitW7fGkiVLsHfvXkRHR+sVS13+CABQ452sarUaw4YNw6uvvlrjezp16gTg7k1FWVlZ2L9/P/bu3Yu9e/ciMTER06ZNw8aNG/WKl0hfLJJEerpz5w4AoLi4GMDd05M2NjbYv38/5HK5Zr3ExMRq762pqLi6ukKhUODcuXMNFHHtqo6U8/LyANy9mWj//v34448/dB5N+vj4QK1WIzs7GwEBAZr2/Px8FBQUwMfH54Gf26FDBxQXFyMsLOyB61pbW2PMmDEYM2YM1Go15syZg3/+859YtGhRtaNZImPiNUkiPahUKnz77bewtrbWFAdLS0vIZDJUVlZq1rt06VKNgwbY29ujoKBAq83CwgLjxo3Drl27ahxyTp8jxNqkpaXV2F51fbFz584AgKioKAghsGTJEp2xjBw5EgA0d6BWWb16NQBo7uitzcSJE3H06FHs37+/2rKCggLNHyO3b9/WWmZhYYHu3bsDQLXHTYiMjUeSRLXYu3cvfvnlFwB3r5ElJSUhOzsbCxcuhEKhAHC3IKxevRoRERGYMmUKbt68iXXr1sHf3x9nzpzR6i8wMBDfffcdVq9eDQ8PD/j5+aFfv35455138O2332LQoEGYPXs2AgICkJeXh+3btyMjIwPOzs71zmXs2LHw8/PDmDFj0KFDB5SUlOC7777Drl270KdPH4wZMwYAEBoaiqeeegoffvghsrOzERERAbVajfT0dISGhmLevHno0aMHoqOj8emnn6KgoACDBg3Cjz/+iI0bN2LcuHEIDQ19YDyvvPIKvv76a4wePRpPP/00AgMDUVJSgrNnz2LHjh24dOkSWrdujZkzZ+KPP/7AkCFD4OnpicuXL2Pt2rXo2bOn1lEsUYOQ9uZaItNU0yMgNjY2omfPnmL9+vVaj2AIcffxio4dOwq5XC66dOkiEhMTRXx8vLj/K/bLL7+IkJAQYWtrKwBoPQ5y+fJlMW3aNOHq6irkcrlo3769mDt3rigvL9eK6f7HRH744QcBQPzwww+15rR582YxadIk0aFDB2FraytsbGzEQw89JN544w2hVCq11r1z54549913RZcuXYS1tbVwdXUVI0aMEJmZmZp1VCqVWLJkifDz8xNWVlbCy8tLxMXFibKyMq2+fHx8xKhRo2qMqaioSMTFxQl/f39hbW0tWrduLfr37y/ee+89UVFRIYQQYseOHWL48OGiTZs2wtraWnh7e4tnn31W5OXl1ZovkTHIhDDSuRwiIqJmhtckiYiIdGCRJCIi0oFFkoiISAcWSSIiIh1YJImIiHSQtEgWFRUhJiYGPj4+sLW1Rf/+/XHixAnNciEE3nzzTbi7u8PW1hZhYWHIzs6WMGIiIjInkg4mMHPmTJw7dw5ffvklPDw8sGnTJoSFheHnn39Gu3btsGrVKnz44YfYuHEj/Pz8sGjRIoSHh+Pnn3+GjY1NnT5DrVbj+vXrcHR0rPM4k0RE1LwIIVBUVAQPDw9YWOhxfCjVA5qlpaXC0tJSfPPNN1rtvXr1Em+88YZQq9XCzc1NvPvuu5plBQUFQi6Xi82bN9f5c65cuVLjvID84Q9/+MMf8/u5cuWKXrVKsiPJO3fuoLKystoRoa2tLTIyMpCTk4MbN25oDX7s5OSEfv364ejRo5g0aVKN/ZaXl2uN5yj+/1gJOTk5cHR0NDhelUqFH374AaGhobCysjK4n6bC3PIFmLM55Gxu+QLMuSrnoqIi+Pn56V0HJB1xp3///rC2tkZSUhLatm2LzZs3Izo6Gv7+/khMTMSAAQNw/fp1uLu7a94zceJEyGQybN26tcY+Fy9eXOPAzElJSbCzs2uwXIiIyHSVlpZiypQpKCws1Iy7XBeSXpP88ssvMX36dLRr1w6Wlpbo1asXJk+ejMzMTIP7jIuL05otXalUwsvLC8OHD9drw9xPpVIhNTUVw4YNM4u/xswtX4A5m0PO5pYvwJyrclYqlQb1JWmR7NChAw4ePIiSkhIolUq4u7vjiSeeQPv27eHm5gbg7vx09x5J5ufno2fPnjr7lMvlWnP6VbGysjLKfxBj9dNUmFu+AHM2B+aWL8CcDc3dJJ6TtLe3h7u7O/7880/s379fM6WPm5ub1hx4SqUSx48fR1BQkITREhEZR6W6EgcuHcDms5tx4NIBVKorH/wmalSSHknu378fQgh07twZv/32G1555RV06dIFzzzzDGQyGWJiYrBs2TJ07NhR8wiIh4cHxo0bZ9Q4hBCaG4l0UalUaNGiBcrKympdr7kwt3wB08rZysoKlpaWksZADSv5fDLm75uPq8qrmjZPhScSIhIQGRApYWR0L0mLZGFhIeLi4nD16lW0atUKUVFRePvttzWHxa+++ipKSkowe/ZsFBQUYODAgdi3b1+dn5Gsi4qKCuTl5aG0tLTW9YQQcHNzw5UrV8zieUtzyxcwrZxlMhk8PT3h4OAgaRzUMJLPJ2PCtgkQ0L5v8pryGiZsm4AdE3ewUJoISYvkxIkTMXHiRJ3LZTIZli5diqVLlzbI56vVauTk5MDS0hIeHh6wtrbW+ctRrVajuLgYDg4O+j2I2kSZW76A6eQshMCtW7dw9epVdOzYkUeUzUyluhLz982vViABQEBABhli9sVgbOexsLTgvpeapEVSahUVFVCr1fDy8nrg4yFqtRoVFRWwsbExi6JhbvkCppWzq6srLl26BJVKxSLZzKTnpmudYr2fgMAV5RWk56ZjsO/gxguMamQev/0eQOpfiET3k/p0LzWcvKI8o65HDYvVgYioEbk7uj94JT3Wo4bFIklE1IiCvYPhqfCEDDWfLZBBBi+FF4K9gxs5MqoJiySZnMOHD6Nbt26wsrKq8+M+ixcvrnWQCan5+vpizZo1UodBJsDSwhIJEQkAUK1QVr1eE7GGN+2YCBbJJujWrVt4/vnn4e3tDblcDjc3N4SHh+Pw4cNSh2YUsbGx6NmzJ3JycrBhwwaj9Hnp0iXIZDLNj7W1Nfz9/bFs2TLoO3yxTCZDSkqKUeIi8xQZEIkdE3egnaKdVrunwpOPf5gYs7671Vgq1ZVIz01HXlEe3B3dEewd3KB/BUZFRaGiogIbN25E+/btkZ+fj7S0NNy+fbvBPrMxXbx4Ec899xw8PT2N3vd3332Hrl27ory8HBkZGZg5cybc3d0xY8YMo38WUW0iAyIxtvPYRv3dQfrjkWQ9JZ9Phm+CL0I3hmJK8hSEbgyFb4Ivks8nN8jnFRQUID09HStXrkRoaCh8fHzQt29fxMXF4bHHHgPw91FTVlaW1vtkMhkOHDigafvf//6H0aNHQ6FQwNHREcHBwbh48aJm+aZNm9CtWzfI5XK4u7tj3rx5Wv3NnDkTrq6uUCgUGDJkCE6fPq1Zfvr0aYSGhsLR0REKhQKBgYE4efIkAODy5csYM2YMWrZsCXt7e3Tt2hV79uzRxH379m1Mnz4dMpkMGzZswIYNG+Ds7Ky1HVJSUgy6A9TFxQVubm7w8fHB1KlTMWDAAJw6dUqz/NSpUxg+fDhat24NJycnDBo0SGu5r68vAGD8+PGQyWSa1wCwa9cu9OnTBzY2NmjdujXGjx+v9dmlpaWYPn06HB0d4e3tjU8//VTv+Kl5sbSwxGDfwZjcbTIG+w5mgTRBLJL1UDVqxv3PPFWNmtEQhdLBwQEODg5ISUnRmjdTX9euXUNISAjkcjm+//57ZGZmYvr06bhz5w4AYP369XjllVcwa9YsnD17Fl9//TX8/f0173/88cdx8+ZN7N27F5mZmejVqxeGDh2KP/74AwAwdepUeHp64sSJE8jMzMTChQs1IynNnTsX5eXlOHToEM6ePYuVK1fCwcEBXl5eyMvLg0KhwJo1a5CXl4cnnniiHlurdidPnkRmZib69eunaSsuLsa0adOQkZGBY8eOoWPHjhg5ciSKiooAACdOnAAAJCYmIi8vT/N69+7dGD9+PEaOHImffvoJaWlp6Nu3r9bnvf/+++jduzd++uknzJkzB88//zwuXLjQYPkRUf3xdKuBpBo1o0WLFtiwYQNmzZqFTz75BL169cKgQYMwadIkdO/evc79rFu3Dk5OTtiyZYumeHXq1Emz/J133sHcuXPx4osvap4j7dOnDwAgIyMDP/74I27evKmZceW9995DSkoKduzYgdmzZyM3N1czFi8AdOzYUdN3bm4uoqKi0K1bNwBA+/btNcvc3Nwgk8ng5OSkmQnGmPr37w8LCwtUVFRApVJh9uzZmDZtmmZ5SEgIFAqFJudPP/0Uzs7OOHjwIEaPHg1XV1cAgLOzs1Z8b7/9NiZNmqQ1l2mPHj20PnvkyJGYM2cOAOC1117DBx98gB9++AGdO3c2ep5EZBw8kjSQPqNmGFtUVBSuX7+Or7/+GhEREThw4AB69eql100uWVlZCA4OrnH6mJs3b+L69esYNGhQje89ffo0iouL4eLiojmydXBwQE5OjuZ0bWxsLGbOnImwsDCsWLFC6zTuiy++iGXLlmHAgAGIj4/HmTNn9NsA9bB161ZkZWXh9OnT2LZtG/773/9i4cKFmuU3b97E7Nmz0bFjRzg5OUGhUKC4uBi5ubm19puVlYWhQ4fWus69f8TIZDK4ubnh5s2b9UuIiBoUi6SBpB41w8bGBsOGDcOiRYtw5MgRPP3004iPjwfw9whC9961qVKptN5va2urs+/algF3T0m6u7sjKytL6+fChQt45ZVXANx9JON///sfRo0ahe+//x4PPfQQdu7cCQCYOXMm/u///g9PPfUUzp49i969e2Pt2rU6P8/CwqLaHaj351NXXl5e8Pf3R0BAAB5//HHExMTg/fffR1lZGQBgzpw5OH36NBISEnDkyBFkZWXBxcUFFRUVtfb7oG0GVJ/PTiaTQa1WG5QHETUOFkkDmdqoGQ899BBKSkoAQHNKMC/v7wJ97008wN2jmvT09BqLjaOjI3x9fXHw4MEaP6tXr164ceMGWrRoAX9/f62f1q1ba9br1KkTFixYgG+//RaRkZFITEzULPPy8sJzzz2H5ORkvPTSS/jss8905ubq6oqioiJNfjXlYyhLS0vcuXNHUwSPHz+OefPmYeTIkejatSvkcjl+//13rfdYWVlVm0qre/fuWnOfElHzwCJpIKlGzbh9+zaGDBmCTZs24cyZM8jJycH27duxatUqjB07FsDdo5pHH30UK1aswPnz53Hw4EH84x//0Opn3rx5UCqVmDRpEk6ePIns7Gx8+eWXmhtJ3nzzTaxbtw5r165FdnY2Tp06pTnaCwsLQ1BQEMaNG4dvv/0Wly5dwpEjR/DGG2/g5MmT+OuvvzBv3jwcOHAAly9fxuHDh3HixAkEBAQAAGJiYrB//37k5OTg1KlT+OGHHzTLatKvXz/Y2dnh9ddfx8WLF5GUlGTw85O3b9/GjRs3cPXqVezduxcJCQkIDQ2FQqEAcPf66KZNm3D+/HkcP34cU6dOrXaU6Ovri7S0NNy4cQN//vknACA+Ph6bN29GfHw8zp8/r7khiYiaNhZJA0k1aoaDgwP69euHDz74ACEhIXj44YexaNEizJo1Cx999JFmvX/961+4c+cOAgMDNZNX38vFxQXff/89iouLMWjQIAQGBuKzzz7TnBKMjo7GO++8g/Xr16Nr164YPXo0srOz7+Ynk2HPnj0ICQnBM888g06dOmHSpEm4fPky2rZtC0tLS9y+fRvTpk1Dp06dMHHiRIwYMUJzU0tlZSXmzp2LgIAAREREoFOnTvj444915tyqVSts2rQJe/bsQbdu3bB582YsXrzYoO0XFhYGd3d3+Pr6Yvbs2Rg5ciS2bt2qWb527VoUFBSgV69eeOqpp/Diiy+iTZs2Wn28//77SE1NhZeXFx555BEAwODBg7F9+3Z8/fXX6NmzJ4YMGYIff/zRoBiJyHTIhL7DjTQxSqUSTk5OKCws1BwtVCkrK0NOTg78/PweOJGzWq2GUqnUuvMRqHl2cS+FF9ZErGnSo2boyrc5M6Wc9fm/WR8qlQp79uzByJEja7yJq7kxt3wB5lyVc221oDZ8BKSeOGoGEVHzxSJpBFWjZhARUfNiHufRiIiIDMAiSUREpAOLJKD3VElEDY3/J4lMg1kXyaq7nkpLSyWOhEhb1eAGlpa8AYxISmZ9446lpSWcnZ0142fa2dnpnH5JrVajoqICZWVlkj8e0BjMLV/AdHJWq9W4desW7Ozs0KKFWX9FiSRn9t/AqpkcHjTQtBACf/31F2xtbQ2ax7CpMbd8AdPK2cLCAt7e3pLHQWTuzL5IymQyuLu7o02bNrUOmq1SqXDo0CGEhISYxQO55pYvYFo5W1tbm80RPJEpM/siWcXS0rLW6z9VA2Hb2NhI/gu0MZhbvoB55kxEteOfqkRERDqwSBIREenAIklERKQDr0lSs1SprjSpQedNLR4iqhtJjyQrKyuxaNEi+Pn5wdbWFh06dMBbb72lNdqIEAJvvvkm3N3dYWtri7CwMM28hkQ1ST6fDN8EX4RuDMWU5CkI3RgK3wRfJJ9PZjxEpBdJi+TKlSuxfv16fPTRRzh//jxWrlyJVatWYe3atZp1Vq1ahQ8//BCffPIJjh8/Dnt7e4SHh6OsrEzCyMlUJZ9PxoRtE7Tm9wSAa8prmLBtQqMXJlOLh4j0I2mRPHLkCMaOHYtRo0bB19cXEyZMwPDhwzUzugshsGbNGvzjH//A2LFj0b17d/z73//G9evXkZKSImXoZIIq1ZWYv28+BKqPe1rVFrMvBpXqSrOMh4j0J+k1yf79++PTTz/Fr7/+ik6dOuH06dPIyMjA6tWrAQA5OTm4ceMGwsLCNO9xcnJCv379cPToUUyaNKlan+Xl5SgvL9e8ViqVAO4+KF7bYAEPUvXe+vTRlDTFfDNyM3C7+DZsLWx1rvN78e84lHMIA70HVltm7JzrG09jaIr7uT7MLV+AOd/fpi+ZkHC6AbVajddffx2rVq2CpaUlKisr8fbbbyMuLg7A3SPNAQMG4Pr163B3d9e8b+LEiZDJZNi6dWu1PhcvXowlS5ZUa09KSoKdnV3DJUNERCartLQUU6ZMQWFhIRQKRZ3fJ+mR5LZt2/Cf//wHSUlJ6Nq1K7KyshATEwMPDw9ER0cb1GdcXBxiY2M1r5VKJby8vDB8+HC9Nsz9VCoVUlNTMWzYMLMYjaUp5puRm4FRSaMeuN7uKbt1HkkaM+f6xtMYmuJ+rg9zyxdgzlU5V51V1JekRfKVV17BwoULNadNu3XrhsuXL2P58uWIjo7WDD6en5+vdSSZn5+Pnj171tinXC6HXC6v1m5lZWWU/yDG6qepaEr5hviFwMXBBdeU12q8DiiDDJ4KT4T4hdT6+IWxcjZWPI2hKe1nYzC3fAHmbGjukt64U1paWm0QZ0tLS6jVagCAn58f3NzckJaWplmuVCpx/PhxBAUFNWqsZPosLSyREJEA4G4BulfV6zURaxqtIJlaPESkP0mL5JgxY/D2229j9+7duHTpEnbu3InVq1dj/PjxAO7O0BETE4Nly5bh66+/xtmzZzFt2jR4eHhg3LhxUoZOJioyIBI7Ju5AO0U7rXZPhSd2TNyByIBIs46HiPQj6enWtWvXYtGiRZgzZw5u3rwJDw8PPPvss3jzzTc167z66qsoKSnB7NmzUVBQgIEDB2Lfvn2wsbGRMHIyZZEBkRjbeazJjHBjavEQUd1JWiQdHR2xZs0arFmzRuc6MpkMS5cuxdKlSxsvMGryLC0sMdh3sNRhaJhaPERUNxzgnIiISAcWSSIiIh1YJImIiHRgkSQiItKB80kSEemg7zygnDe0+WGRJCKqQfL5ZMzfN19rmjNPhScSIhJqfL5V3/WpaeDpViKi++g7DyjnDW2+WCSJiO6h7zygnDe0eWORJCK6R3puerUjwnsJCFxRXkF6brpB61PTwiJJRHSPvKI8vdbTd31qWlgkiYju4e7o/uCV7llP3/WpaWGRJCK6R7B3MDwVntWmN6sigwxeCi8EewcbtD41LSySRET30HceUM4b2ryxSBIR3UffeUA5b2jzxcEEiIhqoO88oJw3tHlikSQi0kHfeUA5b2jzw9OtREREOrBIEhER6cAiSUREpAOLJBERkQ68cYeqqVRXIiM3AwCQkZuBEL8Qye/QM7V5+kwtnuaA2/TBDJnf0tS+y00NiyRpqZoT73bxbWzuvhmjkkbBxcFF0jnxTG2ePlOLpzngNn0wQ+e3NKXvclPE062kYYpz4plaTKYWT3PAbfpgnN9SOiySBMA058QztZhMLZ7mgNv0wTi/pbRYJAmAac6JZ2oxmVo8zUFjbtP7r881lSLB+S2lxSJJAExzTjxTi8nU4mkOGmubJp9Phm+CL0YljQIAjEoaBd8E3yZx2pHzW0qLRZIAmOaceKYWk6nF0xw0xjZt6tfnOL+ltFgkCYBpzolnajGZWjzNQUNv0+ZwfY7zW0qLRZIAmOaceKYWk6nF0xw09DZtDtfnOL+ltCQtkr6+vpDJZNV+5s6dCwAoKyvD3Llz4eLiAgcHB0RFRSE/P1/KkJs1U5wTz9RiMrV4moOG3KbN5foc57eUjqSDCZw4cQKVlX+f5jh37hyGDRuGxx9/HACwYMEC7N69G9u3b4eTkxPmzZuHyMhIHD58WKqQm72qOfEO5RyC8pwSu6fslnyUjsaYp0+fkUk4b6DxNdQ2bU7X5wyd39KUvstNkaRF0tXVVev1ihUr0KFDBwwaNAiFhYX44osvkJSUhCFDhgAAEhMTERAQgGPHjuHRRx+VImSzYGlhiYHeA7Hn3B4M9B5oEl+qhpynz5CRSThvoPE1xDatuj53TXmtxuuSMsjgqfBsMtfnDJnf0tS+y02NyQxLV1FRgU2bNiE2NhYymQyZmZlQqVQICwvTrNOlSxd4e3vj6NGjOotkeXk5ysvLNa+VSiUAQKVSQaVSGRxf1Xvr00dTYi757rqwC0/tfAoCArYWtgAAWwtb/FH8B57a8RQwHhjTeYzEUTYcc9jPCcMS8NTOpwAANhY2AO7uY831uWFroK5UQ12plizGhmQO+/h+NeVsaP4yIUT1P68ksG3bNkyZMgW5ubnw8PBAUlISnnnmGa2CBwB9+/ZFaGgoVq5cWWM/ixcvxpIlS6q1JyUlwc7OrkFiJyIi01ZaWoopU6agsLAQCoWizu8zmSPJL774AiNGjICHh0e9+omLi0NsbKzmtVKphJeXF4YPH67XhrmfSqVCamoqhg0bBisrq3rF2BSYQ74ZuRmah8uBu0cX/3r4X5h+bjr+Uv+lad89ZTcGeg+UIsQGZw77uUqluhJHLh9B0fkiOAY4or9Pf7M4/WhO+7hKTTlXnVXUl0kUycuXL+O7775DcvLfD/W6ubmhoqICBQUFcHZ21rTn5+fDzc1NZ19yuRxyubxau5WVlVH+gxirn6aiOed7o/SGVjGs8pf6L632G6U3mu02qNKc93MVK1gh2C8Ye87vQbBfcLPP937msI/vd2/OhuZuEkUyMTERbdq0wahRf/9VHxgYCCsrK6SlpSEqKgoAcOHCBeTm5iIoKEiqUE0C590zjuZ052NjMGQuw4a+I5nfA+PiNq1O8iKpVquRmJiI6OhotGjxdzhOTk6YMWMGYmNj0apVKygUCrzwwgsICgoy6ztbOe+e8TS3Ox8bkqFzGTbU/1N+D4yP27Rmko+489133yE3NxfTp0+vtuyDDz7A6NGjERUVhZCQELi5uWmdkjU3TX0MSlPDkUnqxtTmMuT3wPi4TXWTvEgOHz4cQgh06tSp2jIbGxusW7cOf/zxB0pKSpCcnFzr9cjmrDmMQWmKODJJ7UxtLkN+D4yP27R2khdJqpvmMAalqYoMiMSl+Zewe8puAHfvZs2Zn2P2BRIwvbkM+T0wPm7T2rFINhHNZQxKU1U1MgkAjkxyD1Oby5DfA+PjNq0di2QTwTsxSQqmNpchvwfGx21aOxbJJoJzxJEUTG0uQ34PjI/btHYskk0E78QkKZjaXIb8Hhgft2ntWCSbEN6JSVIwtbkM+T0wPm5T3SQfTID0w7kMSQqGzmXYUP9P+T0wPm7TmrFINkGcy5CkYMhchg35/5TfA+PjNq2uXqdbKyoqcOHCBdy5c8dY8RAREZkMg4pkaWkpZsyYATs7O3Tt2hW5ubkAgBdeeAErVqwwaoBERERSMahIxsXF4fTp0zhw4ABsbGw07WFhYdi6davRgiMiIpKSQdckU1JSsHXrVjz66KOQyf6+Zbhr1664ePGi0YIjIiKSkkFF8tatW2jTpk219pKSEq2iSUREdK+mNmelQUWyd+/e2L17N1544QUA0BTGzz//3OwnRCYiopo1xTkrDSqS77zzDkaMGIGff/4Zd+7cQUJCAn7++WccOXIEBw8eNHaMRETUxFXNWXn/lFxVc1aa6qAFBt24M3DgQJw+fRp37txBt27d8O2336JNmzY4evQoAgMDjR0jERE1YU15zkq9jyRVKhWeffZZLFq0CJ999llDxERERM2IPnNWmtpgBnofSVpZWeGrr75qiFiIiKgZaspzVhp0unXcuHFISUkxcihERNQcNeU5Kw26cadjx45YunQpDh8+jMDAQNjb22stf/HFF40SHBERNX1Vc1ZeU16r8bqkDDJ4KjxNcs5Kg4rkF198AWdnZ2RmZiIzM1NrmUwmY5EkIiKNqjkrJ2ybABlkWoXS1OesNKhI5uTkGDsOIiJqxqrmrKzpOck1EWtM8vEPwAhTZQlx9y8CjrRDRES1aYpzVho8Vda///1vdOvWDba2trC1tUX37t3x5ZdfGjM2IiJqZqrmrJzcbTIG+w426QIJGHgkuXr1aixatAjz5s3DgAEDAAAZGRl47rnn8Pvvv2PBggVGDZKIiEgKBhXJtWvXYv369Zg2bZqm7bHHHkPXrl2xePFiFkkiImoWDDrdmpeXh/79+1dr79+/P/LyTO9hUCIiIkMYVCT9/f2xbdu2au1bt25Fx44d6x0UERGRKTDodOuSJUvwxBNP4NChQ5prkocPH0ZaWlqNxZOI9NPU5twjaq4MOpKMiorC8ePH0bp1a6SkpCAlJQWtW7fGjz/+iPHjx+vV17Vr1/Dkk0/CxcUFtra26NatG06ePKlZLoTAm2++CXd3d9ja2iIsLAzZ2dmGhE3UJCSfT4Zvgi9CN4ZiSvIUhG4MhW+CL5LPJ0sdGpHZMfg5ycDAQGzatKleH/7nn39iwIABCA0Nxd69e+Hq6ors7Gy0bNlSs86qVavw4YcfYuPGjfDz88OiRYsQHh6On3/+GTY2NvX6fCJT01Tn3CNqrgwqknv27IGlpSXCw8O12vfv3w+1Wo0RI0bUqZ+VK1fCy8sLiYmJmjY/Pz/Nv4UQWLNmDf7xj39g7NixAO4+n9m2bVukpKRg0qRJhoRPZJIeNOeeDDLE7IvB2M5jeeqVqJEYVCQXLlyIFStWVGsXQmDhwoV1LpJff/01wsPD8fjjj+PgwYNo164d5syZg1mzZgG4O/zdjRs3EBYWpnmPk5MT+vXrh6NHj9ZYJMvLy1FeXq55rVQqAdydB1OlUumV572q3lufPpoSc8sXkD7njNwM3C6+DVsLW53r/F78Ow7lHMJA74FG+Uypc25s5pYvwJzvb9OXTFSNK6cHW1tbnD9/Hr6+vlrtly5dQteuXVFSUlKnfqpOl8bGxuLxxx/HiRMnMH/+fHzyySeIjo7GkSNHMGDAAFy/fh3u7n9PoTJx4kTIZDJs3bq1Wp+LFy/GkiVLqrUnJSXBzs5OjyyJiKi5KC0txZQpU1BYWAiFQlHn9xl0JOnk5IT/+7//q1Ykf/vtt2rTZtVGrVajd+/eeOeddwAAjzzyCM6dO6cpkoaIi4tDbGys5rVSqYSXlxeGDx+u14a5n0qlQmpqKoYNGwYrKyuD+2kqzC1fQPqcM3IzMCpp1APX2z1lt1GPJM1pP5tbvgBzrsq56qyivgwqkmPHjkVMTAx27tyJDh06ALhbIF966SU89thjde7H3d0dDz30kFZbQEAAvvrqKwCAm5sbACA/P1/rSDI/Px89e/assU+5XA65XF6t3crKyij/QYzVT1NhbvkC0uUc4hcCFweXB865F+IXYvRrkua2n80tX4A5G5q7QY+ArFq1Cvb29ujSpQv8/Pzg5+eHLl26wMXFBe+9916d+xkwYAAuXLig1fbrr7/Cx8cHwN2beNzc3JCWlqZZrlQqcfz4cQQFBRkSOpHJqppzD/h7jr0qpj7nHlFzZfDp1iNHjiA1NRWnT5+Gra0tevTogeBg/WaVXrBgAfr374933nkHEydOxI8//ohPP/0Un376KYC702/FxMRg2bJl6Nixo+YREA8PD4wbN86Q0IlMWlOdc4+oudKrSB49ehS3b9/G6NGjIZPJMHz4cOTl5SE+Ph6lpaUYN24c1q5dW+Ppzpr06dMHO3fuRFxcHJYuXQo/Pz+sWbMGU6dO1azz6quvoqSkBLNnz0ZBQQEGDhyIffv28RlJaraa4px7RM2VXkVy6dKlGDx4MEaPHg0AOHv2LGbNmoXo6GgEBATg3XffhYeHBxYvXlznPkePHq3pryYymQxLly7F0qVL9QmVqEmrmnOPiKSl1zXJrKwsDB06VPN6y5Yt6Nu3Lz777DPExsbiww8/5NitRETUbOhVJP/880+0bdtW8/rgwYNaAwf06dMHV65cMV50REREEtKrSLZt2xY5OTkAgIqKCpw6dQqPPvqoZnlRUVGzvMW4Ul2JjNwMAHefZatUV0ocERERNQa9iuTIkSOxcOFCpKenIy4uDnZ2dlp3tJ45c0bz3GRzUTUjQ9VD3qOSRnFGBiIiM6HXjTtvvfUWIiMjMWjQIDg4OGDjxo2wtrbWLP/Xv/6F4cOHGz1Iqdw7I8O942kae0aGhp470BT753yJxqfvNr3/DElDDFJA1NTpVSRbt26NQ4cOobCwEA4ODrC01P5Cbd++HQ4ODkYNUCqNNSND8vnkGp+JS4hIMEoBNsX+Gzomc6TvNq1a/3bxbWzuvhmjkkbBxcGF+4DoPgaNuOPk5FStQAJAq1attI4sm7L03HStXzj3ExC4oryC9Nx0gz+j6kj1/s+pOlKt7yldU+y/oWMyR/puU+4DorozqEiag7yiPKOud78HHakCQMy+GINvEjLF/hs6JnOk7zblPiDSD4ukDu6O7g9eSY/17tfQR6qm2H9jHJ2bG323KfcBkX5YJHUI9g6Gp8Kz2kDTVWSQwUvhhWBv/carrdLQR6qm2H9Dx2SO9N2m3AdE+mGR1KGhZ2Ro6CNVU+y/oWMyR/puU+4DIv2wSNaiakaGdop2Wu2eCs96P/7R0Eeqpth/Q8dkjvTdptwHRPphkXyAyIBIXJp/Cbun7AZwd1b4nPk59b5NvqGPVE2xf86XaHz6blPuAyL9sEjWgaWFJQZ6DwQADPQeaLRfIA15pGqq/Td0TOZI323KfUBUdwZNukzG09BzB5pi/5wv0fj03aZV6x/KOQTlOSV2T9nNEXeIasAiaQIaeu5AU+yf8yUan77btOoMyZ5ze4x6hoSoOeHpViIiIh1YJImIiHRgkSQiItKBRZKIiEgHFkkiIiIdWCSJiIh0YJEkIiLSgUWSiIhIBxZJIiIiHVgkiYiIdGCRJCIi0oFFkoiISAcWSSIiIh1YJImIiHSQtEguXrwYMplM66dLly6a5WVlZZg7dy5cXFzg4OCAqKgo5OfnSxgxERGZE8mPJLt27Yq8vDzNT0ZGhmbZggULsGvXLmzfvh0HDx7E9evXERnJWdOJiKhxSD7pcosWLeDm5latvbCwEF988QWSkpIwZMgQAEBiYiICAgJw7NgxPProo40dKhERmRnJi2R2djY8PDxgY2ODoKAgLF++HN7e3sjMzIRKpUJYWJhm3S5dusDb2xtHjx7VWSTLy8tRXl6uea1UKgEAKpUKKpXK4Dir3lufPpoSc8sXYM7mwNzyBZjz/W36kgkhhFGiMsDevXtRXFyMzp07Iy8vD0uWLMG1a9dw7tw57Nq1C88884xWwQOAvn37IjQ0FCtXrqyxz8WLF2PJkiXV2pOSkmBnZ9cgeRARkWkrLS3FlClTUFhYCIVCUef3SVok71dQUAAfHx+sXr0atra2BhXJmo4kvby88Pvvv+u1Ye6nUqmQmpqKYcOGwcrKyuB+mgpzyxdgzuaQs7nlCzDnqpyVSiVat26td5GU/HTrvZydndGpUyf89ttvGDZsGCoqKlBQUABnZ2fNOvn5+TVew6wil8shl8urtVtZWRnlP4ix+mkqzC1fgDmbA3PLF2DOhuYu+d2t9youLsbFixfh7u6OwMBAWFlZIS0tTbP8woULyM3NRVBQkIRREhGRuZD0SPLll1/GmDFj4OPjg+vXryM+Ph6WlpaYPHkynJycMGPGDMTGxqJVq1ZQKBR44YUXEBQUxDtbiYioUUhaJK9evYrJkyfj9u3bcHV1xcCBA3Hs2DG4uroCAD744ANYWFggKioK5eXlCA8Px8cffyxlyEREZEYkLZJbtmypdbmNjQ3WrVuHdevWNVJEREREfzOpa5JERESmhEWSiIhIBxZJIiIiHVgkiYiIdGCRJCIi0oFFkoiISAcWSSIiIh1YJImIiHRgkSQiItKBRZKIiEgHFkkiIiIdWCSJiIh0MKlJl5uLSnUl0nPTkVeUB3dHdwR7B8PSwlLqsIiImhypf5+ySBpZ8vlkzN83H1eVVzVtngpPJEQkIDIgUsLIiIiaFlP4fcrTrUaUfD4ZE7ZN0NqhAHBNeQ0Ttk1A8vlkiSIjImpaTOX3KYukkVSqKzF/33wIiGrLqtpi9sWgUl3Z2KERETUppvT7lEXSSNJz06v9xXMvAYEryitIz01vxKiIiJoeU/p9yiJpJHlFeUZdj4jIXJnS71MWSSNxd3Q36npERObKlH6fskgaSbB3MDwVnpBBVuNyGWTwUngh2Du4kSMjImpaTOn3KYukkVhaWCIhIgEAqu3YqtdrItbweUkiogcwpd+nLJJGFBkQiR0Td6Cdop1Wu6fCEzsm7uBzkkREdWQqv085mICRRQZEYmznsRxxh4ionkzh9ymLZAOwtLDEYN/BUodBRNTkSf37lKdbiYiIdGCRJCIi0oFFkoiISAcWSSIiIh1YJImIiHQwmSK5YsUKyGQyxMTEaNrKysowd+5cuLi4wMHBAVFRUcjPz5cuSCIiMismUSRPnDiBf/7zn+jevbtW+4IFC7Br1y5s374dBw8exPXr1xEZyQfyiYiocUheJIuLizF16lR89tlnaNmypaa9sLAQX3zxBVavXo0hQ4YgMDAQiYmJOHLkCI4dOyZhxEREZC4kH0xg7ty5GDVqFMLCwrBs2TJNe2ZmJlQqFcLCwjRtXbp0gbe3N44ePYpHH320xv7Ky8tRXl6uea1UKgEAKpUKKpXK4Dir3lufPpoSc8sXYM7mwNzyBZjz/W36krRIbtmyBadOncKJEyeqLbtx4wasra3h7Oys1d62bVvcuHFDZ5/Lly/HkiVLqrV/++23sLOzq3fMqamp9e6jKTG3fAHmbA7MLV+AOZeWlhrUh2RF8sqVK5g/fz5SU1NhY2NjtH7j4uIQGxurea1UKuHl5YXhw4dDoVAY3K9KpUJqaiqGDRsGKysrY4Rq0swtX4A5m0PO5pYvwJyrcq46q6gvyYpkZmYmbt68iV69emnaKisrcejQIXz00UfYv38/KioqUFBQoHU0mZ+fDzc3N539yuVyyOXyau1WVlZG+Q9irH6aCnPLF2DO5sDc8gWYs6G5S1Ykhw4dirNnz2q1PfPMM+jSpQtee+01eHl5wcrKCmlpaYiKigIAXLhwAbm5uQgKCpIiZCIiMjOSFUlHR0c8/PDDWm329vZwcXHRtM+YMQOxsbFo1aoVFAoFXnjhBQQFBem8aYeIiMiYJL+7tTYffPABLCwsEBUVhfLycoSHh+Pjjz+WOiwiIjITJlUkDxw4oPXaxsYG69atw7p166QJiIiIzJrkgwkQERGZKhZJIiIiHVgkiYiIdGCRJCIi0sGkbtwhIqKmo1JdifTcdOQV5cHd0R3B3sGwtLCUOiyjYpEkIiK9JZ9Pxvx983FVeVXT5qnwREJEAiIDms+UhjzdSkREekk+n4wJ2yZoFUgAuKa8hgnbJiD5fLJEkRkfiyQREdVZpboS8/fNh4CotqyqLWZfDCrVlY0dWoNgkSQiojpLz02vdgR5LwGBK8orSM9Nb8SoGg6LJBER1VleUZ5R1zN1LJJERFRn7o7uRl3P1LFIEhFRnQV7B8NT4QkZZDUul0EGL4UXgr2DGzmyhsEiSUREdWZpYYmEiAQAqFYoq16viVjTbJ6XZJEkIiK9RAZEYsfEHWinaKfV7qnwxI6JO5rVc5IcTICIiPQWGRCJsZ3HcsQdIiKimlhaWGKw72Cpw2hQPN1KRESkA4skERGRDiySREREOjT7a5JC3B1LUKlU1qsflUqF0tJSKJVKWFlZGSM0k2Zu+QLM2RxyNrd8AeZclXNVDaiqCXXV7ItkUVERAMDLy0viSIiISGpFRUVwcnKq8/oyoW9ZbWLUajWuX78OR0dHyGQ1jxBRF0qlEl5eXrhy5QoUCoURIzRN5pYvwJzNIWdzyxdgzlU5CyFQVFQEDw8PWFjU/Upjsz+StLCwgKenp9H6UygUZvMfDTC/fAHmbA7MLV+AOQPQ6wiyCm/cISIi0oFFkoiISAcWyTqSy+WIj4+HXC6XOpRGYW75AszZHJhbvgBzrq9mf+MOERGRoXgkSUREpAOLJBERkQ4skkRERDqwSBIREenAIlkH69atg6+vL2xsbNCvXz/8+OOPUofUYBYvXgyZTKb106VLF6nDMqpDhw5hzJgx8PDwgEwmQ0pKitZyIQTefPNNuLu7w9bWFmFhYcjOzpYmWCN4UL5PP/10tX0eEREhTbBGsnz5cvTp0weOjo5o06YNxo0bhwsXLmitU1ZWhrlz58LFxQUODg6IiopCfn6+RBHXT13yHTx4cLX9/Nxzz0kUcf2tX78e3bt31wwYEBQUhL1792qWG2v/skg+wNatWxEbG4v4+HicOnUKPXr0QHh4OG7evCl1aA2ma9euyMvL0/xkZGRIHZJRlZSUoEePHli3bl2Ny1etWoUPP/wQn3zyCY4fPw57e3uEh4ejrKyskSM1jgflCwARERFa+3zz5s2NGKHxHTx4EHPnzsWxY8eQmpoKlUqF4cOHo6SkRLPOggULsGvXLmzfvh0HDx7E9evXERkZKWHUhqtLvgAwa9Ysrf28atUqiSKuP09PT6xYsQKZmZk4efIkhgwZgrFjx+J///sfACPuX0G16tu3r5g7d67mdWVlpfDw8BDLly+XMKqGEx8fL3r06CF1GI0GgNi5c6fmtVqtFm5ubuLdd9/VtBUUFAi5XC42b94sQYTGdX++QggRHR0txo4dK0k8jeXmzZsCgDh48KAQ4u4+tbKyEtu3b9esc/78eQFAHD16VKowjeb+fIUQYtCgQWL+/PnSBdUIWrZsKT7//HOj7l8eSdaioqICmZmZCAsL07RZWFggLCwMR48elTCyhpWdnQ0PDw+0b98eU6dORW5urtQhNZqcnBzcuHFDa587OTmhX79+zXqfHzhwAG3atEHnzp3x/PPP4/bt21KHZFSFhYUAgFatWgEAMjMzoVKptPZzly5d4O3t3Sz28/35VvnPf/6D1q1b4+GHH0ZcXBxKS0ulCM/oKisrsWXLFpSUlCAoKMio+7fZD3BeH7///jsqKyvRtm1brfa2bdvil19+kSiqhtWvXz9s2LABnTt3Rl5eHpYsWYLg4GCcO3cOjo6OUofX4G7cuAEANe7zqmXNTUREBCIjI+Hn54eLFy/i9ddfx4gRI3D06FFYWlpKHV69qdVqxMTEYMCAAXj44YcB3N3P1tbWcHZ21lq3OeznmvIFgClTpsDHxwceHh44c+YMXnvtNVy4cAHJyckSRls/Z8+eRVBQEMrKyuDg4ICdO3fioYceQlZWltH2L4skaRkxYoTm3927d0e/fv3g4+ODbdu2YcaMGRJGRg1l0qRJmn9369YN3bt3R4cOHXDgwAEMHTpUwsiMY+7cuTh37lyzu7aui658Z8+erfl3t27d4O7ujqFDh+LixYvo0KFDY4dpFJ07d0ZWVhYKCwuxY8cOREdH4+DBg0b9DJ5urUXr1q1haWlZ7Y6o/Px8uLm5SRRV43J2dkanTp3w22+/SR1Ko6jar+a8z9u3b4/WrVs3i30+b948fPPNN/jhhx+0psxzc3NDRUUFCgoKtNZv6vtZV7416devHwA06f1sbW0Nf39/BAYGYvny5ejRowcSEhKMun9ZJGthbW2NwMBApKWladrUajXS0tIQFBQkYWSNp7i4GBcvXoS7u7vUoTQKPz8/uLm5ae1zpVKJ48ePm80+v3r1Km7fvt2k97kQAvPmzcPOnTvx/fffw8/PT2t5YGAgrKystPbzhQsXkJub2yT384PyrUlWVhYANOn9fD+1Wo3y8nLj7l/j3lvU/GzZskXI5XKxYcMG8fPPP4vZs2cLZ2dncePGDalDaxAvvfSSOHDggMjJyRGHDx8WYWFhonXr1uLmzZtSh2Y0RUVF4qeffhI//fSTACBWr14tfvrpJ3H58mUhhBArVqwQzs7O4r///a84c+aMGDt2rPDz8xN//fWXxJEbprZ8i4qKxMsvvyyOHj0qcnJyxHfffSd69eolOnbsKMrKyqQO3WDPP/+8cHJyEgcOHBB5eXman9LSUs06zz33nPD29hbff/+9OHnypAgKChJBQUESRm24B+X722+/iaVLl4qTJ0+KnJwc8d///le0b99ehISESBy54RYuXCgOHjwocnJyxJkzZ8TChQuFTCYT3377rRDCePuXRbIO1q5dK7y9vYW1tbXo27evOHbsmNQhNZgnnnhCuLu7C2tra9GuXTvxxBNPiN9++03qsIzqhx9+EACq/URHRwsh7j4GsmjRItG2bVshl8vF0KFDxYULF6QNuh5qy7e0tFQMHz5cuLq6CisrK+Hj4yNmzZrV5P8IrClfACIxMVGzzl9//SXmzJkjWrZsKezs7MT48eNFXl6edEHXw4Pyzc3NFSEhIaJVq1ZCLpcLf39/8corr4jCwkJpA6+H6dOnCx8fH2FtbS1cXV3F0KFDNQVSCOPtX06VRUREpAOvSRIREenAIklERKQDiyQREZEOLJJEREQ6sEgSERHpwCJJRESkA4skERGRDiySREREOrBIEpm4DRs2VJvypyl4+umnMW7cOKnDIKoXFkmiOnj66achk8k0Py4uLoiIiMCZM2f06mfx4sXo2bNnwwR5j0uXLkEmk6FNmzYoKirSWtazZ08sXry4wWMgag5YJInqKCIiAnl5ecjLy0NaWhpatGiB0aNHSx1WrYqKivDee+9JHYbRCCFw584dqcMgM8IiSVRHcrkcbm5ucHNzQ8+ePbFw4UJcuXIFt27d0qzz2muvoVOnTrCzs0P79u2xaNEiqFQqAHdPmy5ZsgSnT5/WHJFu2LABAFBQUIBnn30Wbdu2hY2NDR5++GF88803Wp+/f/9+BAQEwMHBQVOwH+SFF17A6tWrcfPmTZ3ryGQypKSkaLU5OztrYqs6Kt22bRuCg4Nha2uLPn364Ndff8WJEyfQu3dvODg4YMSIEVrbosqSJUvg6uoKhUKB5557DhUVFZplarUay5cvh5+fH2xtbdGjRw/s2LFDs/zAgQOQyWTYu3cvAgMDIZfLzWbyZDINLaQOgKgpKi4uxqZNm+Dv7w8XFxdNu6OjIzZs2AAPDw+cPXsWs2bNgqOjI1599VU88cQTOHfuHPbt24fvvvsOAODk5AS1Wo0RI0agqKgImzZtQocOHfDzzz/D0tJS029paSnee+89fPnll7CwsMCTTz6Jl19+Gf/5z39qjXPy5MlITU3F0qVL8dFHH9Ur5/j4eKxZswbe3t6YPn06pkyZAkdHRyQkJMDOzg4TJ07Em2++ifXr12vek5aWBhsbGxw4cACXLl3CM888AxcXF7z99tsAgOXLl2PTpk345JNP0LFjRxw6dAhPPvkkXF1dMWjQIE0/CxcuxHvvvYf27dujZcuW9cqDSC9Gm7eEqBmLjo4WlpaWwt7eXtjb2wsAwt3dXWRmZtb6vnfffVcEBgZqXsfHx4sePXporbN//35hYWGhczquxMREAUBryrJ169aJtm3b6vzcnJwcAUD89NNPYt++fcLKykrz/h49eoj4+HjNugDEzp07td7v5OSkmWapqq/PP/9cs3zz5s0CgEhLS9O0LV++XHTu3FnzOjo6WrRq1UqUlJRo2tavXy8cHBxEZWWlKCsrE3Z2duLIkSNanz1jxgwxefJkIcTf03ylpKTozJWoIfFIkqiOQkNDNUdJf/75Jz7++GOMGDECP/74I3x8fAAAW7duxYcffoiLFy+iuLgYd+7cgUKhqLXfrKwseHp6olOnTjrXsbOzQ4cOHTSv3d3daz2Feq/w8HAMHDgQixYtQlJSUp3eU5Pu3btr/t22bVsAQLdu3bTa7o+pR48esLOz07wOCgpCcXExrly5guLiYpSWlmLYsGFa76moqMAjjzyi1da7d2+D4yaqDxZJojqyt7eHv7+/5vXnn38OJycnfPbZZ1i2bBmOHj2KqVOnYsmSJQgPD4eTkxO2bNmC999/v9Z+bW1tH/jZVlZWWq9lMhmEHlPBrlixAkFBQXjllVeqLaupr6rrqLpikMlkNbap1eo6x1RcXAwA2L17N9q1a6e1TC6Xa722t7evc79ExsQiSWQgmUwGCwsL/PXXXwCAI0eOwMfHB2+88YZmncuXL2u9x9raGpWVlVpt3bt3x9WrV/Hrr7/WejRZH3379kVkZCQWLlxYbZmrq6vWTUDZ2dkoLS01yueePn0af/31l+YPgWPHjsHBwQFeXl5o1aoV5HI5cnNzta4/EpkSFkmiOiovL8eNGzcA3D3d+tFHH6G4uBhjxowBAHTs2BG5ubnYsmUL+vTpg927d2Pnzp1affj6+iInJ0dzitXR0RGDBg1CSEgIoqKisHr1avj7++OXX36BTCZDRESE0eJ/++230bVrV7Roof21HzJkCD766CMEBQWhsrISr732WrUjV0NVVFRgxowZ+Mc//oFLly4hPj4e8+bNg4WFBRwdHfHyyy9jwYIFUKvVGDhwIAoLC3H48GEoFApER0cbJQai+uAjIER1tG/fPri7u8Pd3R39+vXDiRMnsH37dgwePBgA8Nhjj2HBggWYN28eevbsiSNHjmDRokVafURFRSEiIgKhoaFwdXXF5s2bAQBfffUV+vTpg8mTJ+Ohhx7Cq6++Wu2Is746deqE6dOno6ysTKv9/fffh5eXF4KDgzFlyhS8/PLLWtcR62Po0KHo2LEjQkJC8MQTT+Cxxx7TGsjgrbfewqJFi7B8+XIEBAQgIiICu3fvhp+fn1E+n6i+ZEKfCxtERERmhEeSREREOrBIEhER6cAiSUREpAOLJBERkQ4skkRERDqwSBIREenAIklERKQDiyQREZEOLJJEREQ6sEgSERHpwCJJRESkw/8Dr/YMdZSGXVcAAAAASUVORK5CYII=",
            "text/plain": [
              "<Figure size 500x300 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "trial_logs = compiled_program.trial_logs\n",
        "\n",
        "# Extracting trial numbers, scores, and pruning status\n",
        "trial_numbers = list(trial_logs.keys())\n",
        "scores = [trial_logs[trial]['score'] for trial in trial_numbers]\n",
        "pruning_status = [trial_logs[trial]['pruned'] for trial in trial_numbers]\n",
        "\n",
        "# Plot setup\n",
        "plt.figure(figsize=(5, 3))\n",
        "\n",
        "# Plotting each point\n",
        "for trial_number, score, pruned in zip(trial_numbers, scores, pruning_status):\n",
        "    if pruned:\n",
        "        plt.scatter(trial_number, score, color='grey', label='Pruned Batch' if 'Pruned Batch' not in plt.gca().get_legend_handles_labels()[1] else \"\")\n",
        "    else:\n",
        "        plt.scatter(trial_number, score, color='green', label='Successful Batch' if 'Successful Batch' not in plt.gca().get_legend_handles_labels()[1] else \"\")\n",
        "\n",
        "plt.xlabel('Batch Number')\n",
        "plt.ylabel('Score')\n",
        "plt.title('Batch Scores')\n",
        "plt.grid(True)\n",
        "plt.legend()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5_Rwxfa1qZH4"
      },
      "source": [
        "We can also __visualize the best prompts__ discovered by MIPRO as our trials progress... (though note that score increases are also due to the selected fewshot examples, which are not shown here for conciseness)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "NnARfPRHqZH4"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Baseline program | Score: 0:\n",
            "Prompt 1 Instruction: context, question -> answer\n",
            "\n",
            "----------------\n",
            "Best program after 0 batches | Score: 56.5:\n",
            "Prompt 1 Instruction: Provide a yes or no answer based on the given context and question. The context and question are used as input to generate the answer.\n",
            "\n",
            "Best program after 5 batches | Score: 56.5:\n",
            "Prompt 1 Instruction: Provide a yes or no answer based on the given context and question. The context and question are used as input to generate the answer.\n",
            "\n",
            "Best program after 10 batches | Score: 69.5:\n",
            "Prompt 1 Instruction: Given the context and question, generate a logical answer based on the reasoning process.\n",
            "\n",
            "Best program after 15 batches | Score: 69.5:\n",
            "Prompt 1 Instruction: Given the context and question, generate a logical answer based on the reasoning process.\n",
            "\n",
            "Best program after 20 batches | Score: 71.0:\n",
            "Prompt 1 Instruction: Determine if it can be logically concluded that a kayak is nearby based on the given context and question, and generate a yes or no answer using the language model.\n",
            "\n",
            "Best program after 25 batches | Score: 71.0:\n",
            "Prompt 1 Instruction: Determine if it can be logically concluded that a kayak is nearby based on the given context and question, and generate a yes or no answer using the language model.\n",
            "\n"
          ]
        }
      ],
      "source": [
        "best_score = 0\n",
        "\n",
        "def get_signature(predictor):\n",
        "    if (hasattr(predictor, 'extended_signature')):\n",
        "        return predictor.extended_signature\n",
        "    elif (hasattr(predictor, 'signature')):\n",
        "        return predictor.signature\n",
        "\n",
        "print(f\"Baseline program | Score: {best_score}:\")\n",
        "for i,predictor in enumerate(program.predictors()):\n",
        "    print(f\"Prompt {i+1} Instruction: {get_signature(predictor).instructions}\")\n",
        "print()\n",
        "\n",
        "print(\"----------------\")\n",
        "\n",
        "for trial_num in compiled_program.trial_logs:\n",
        "    program_score = compiled_program.trial_logs[trial_num][\"score\"]\n",
        "    program_pruned = compiled_program.trial_logs[trial_num][\"pruned\"]\n",
        "    if program_score > best_score and not program_pruned and compiled_program.trial_logs[trial_num][\"full_eval\"]:\n",
        "        best_score = program_score\n",
        "        best_program_so_far = compiled_program.trial_logs[trial_num][\"program\"]\n",
        "    if trial_num % 5 == 0:\n",
        "        print(f\"Best program after {trial_num} batches | Score: {best_score}:\")\n",
        "        for i,predictor in enumerate(best_program_so_far.predictors()):\n",
        "            print(f\"Prompt {i+1} Instruction: {get_signature(predictor).instructions}\")\n",
        "        print()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### 3d] Saving your program for later use\n",
        "\n",
        "Now that we've gone through all this work of compiling a program it would be a shame to throw it away.  Fortunately we don't have to.  We can save your compiled program to disk with .save()!\n",
        "\n",
        "This file is also human interpretable, so it's worth taking a look at the optimized program.  You can load it later with .load() on a program with the same modules."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {},
      "outputs": [],
      "source": [
        "compiled_program.save(\"compiled_program.dspy\")"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "dspy_test",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.7"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "037b10adf9724e058c6468f4ca74ed86": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_942897aa22214745adee56d2c54447e9",
            "placeholder": "​",
            "style": "IPY_MODEL_0b49910850f3445abe63b6bc7ac18412",
            "value": " 46.2M/46.2M [00:00&lt;00:00, 62.2MB/s]"
          }
        },
        "09b3f2c456af41cba9264dbb9a724027": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0a16c7f37b9a4bbe9bfe7e737ea62801": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "0a1744c363f640b5b8939f32f6e72922": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0b49910850f3445abe63b6bc7ac18412": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "12a9763cd97742f4ab80c0494a398ca6": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_9d4f14862b704d9bb53d9e74365d14ac",
              "IPY_MODEL_56989c76e1534cfd8c9b0da93b3b8bf8",
              "IPY_MODEL_f120c447645446e4a04791b97ce9ae93"
            ],
            "layout": "IPY_MODEL_6a5d5ea44c0d4e4384b8956b48c34f14"
          }
        },
        "134fa83980e142bf82d5b97cfc10da69": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "19fe27a0df5a4d39873dd1031904405c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "1aab1d48c6124526bee6c74892ecd953": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1bb155669e2a441d8992030e8aaa84a9": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "1c7d71e42c8c494f867b30d781beb681": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "1f23a95e292249a19055f70cda2622a3": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_33267de625db44c2a63d7c7d412a7e61",
            "max": 7405,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_19fe27a0df5a4d39873dd1031904405c",
            "value": 7405
          }
        },
        "20d8df0ee0a4442582686846757bcc7f": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "287c9f993cd44039923fdc122cd9e040": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "29cab51fec0b4dacaa8f829ff217c839": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2a75d45acfb44463af974acb7a1b0d8e": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2d47a6a66054438d8e9a3c5e5767c056": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2f1e652cfa514054abfda794bdfa61a5": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "33267de625db44c2a63d7c7d412a7e61": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "348bb4fff900492cba8333156626a947": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_697eaeb1a004493a9260a3a89671a9ca",
              "IPY_MODEL_8a64950ea896468da3d10f46e9718ec8",
              "IPY_MODEL_7b0b45020d0f45288e2f0e4ceff22524"
            ],
            "layout": "IPY_MODEL_ad21e20d1f1b4b6ea74fdab03af8acdb"
          }
        },
        "35e4e99036894a2ba37d9ba23581a8ff": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "38abb8f460d24c27a66c54bb0417f8ce": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "398d7d36bbd041fe81ec633e973fc504": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_41961130caaf4537a4c1793574c785d7",
            "placeholder": "​",
            "style": "IPY_MODEL_e9935b60c48d46469904492e20f9c2e8",
            "value": "Generating validation split: 100%"
          }
        },
        "3b57d0a9768f4befb7509581289035ad": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "41961130caaf4537a4c1793574c785d7": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "43ed7af1d9c84ac8a6ec2195e48e60eb": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4cd1dc71c80b401da19ed52ef5148d60": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_de6507e9f45741218bf31a9af728b2bf",
            "placeholder": "​",
            "style": "IPY_MODEL_38abb8f460d24c27a66c54bb0417f8ce",
            "value": "Generating test split: 100%"
          }
        },
        "52f1714412df4ec78f6ff7d0f8a69862": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_988055b44cf9411e8d45d8a4185fef07",
              "IPY_MODEL_59be08d44c824d76b8525df14191d569",
              "IPY_MODEL_6fde64f36dd84d8eae670efe95e2313a"
            ],
            "layout": "IPY_MODEL_66e6502a463145daa440e967d8bdfdca"
          }
        },
        "531c380db44a404ebb168648ea77c3f4": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_74d58314b1c449e0b7dec3bda8b653c2",
            "placeholder": "​",
            "style": "IPY_MODEL_8b7b2b9489ae49049befae848d0fb1a1",
            "value": "Downloading builder script: 100%"
          }
        },
        "567646442eb940b09456328d49248945": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "56989c76e1534cfd8c9b0da93b3b8bf8": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_9a733957f8fc446fbde053ba87289c71",
            "max": 90447,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_287c9f993cd44039923fdc122cd9e040",
            "value": 90447
          }
        },
        "56b80fc45dd247deadceffe17557f0f9": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "598c6f37331e448889b175393a00deff": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "59be08d44c824d76b8525df14191d569": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a105ef4f1f754c44b7bc0ec8edf0c0cc",
            "max": 47454698,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_1c7d71e42c8c494f867b30d781beb681",
            "value": 47454698
          }
        },
        "5cfebfd6308349038f9cd7ad5bc00fe5": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6028a5e08f8641f5b8e8182e50f55419": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_09b3f2c456af41cba9264dbb9a724027",
            "placeholder": "​",
            "style": "IPY_MODEL_e2f3a3377ad64a4cb9f3a42c1ac97344",
            "value": " 7405/7405 [00:02&lt;00:00, 3033.88 examples/s]"
          }
        },
        "62d8196169bb4a76a94c16d6f1ca61a6": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "66e6502a463145daa440e967d8bdfdca": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "69412c7605ec48859baef6f31c91c520": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_dba7ce7c756d468b94d0bc8f4815ab6f",
            "placeholder": "​",
            "style": "IPY_MODEL_926cbb119ea745cf9e344c0e50b75f62",
            "value": " 7405/7405 [00:04&lt;00:00, 1868.04 examples/s]"
          }
        },
        "697eaeb1a004493a9260a3a89671a9ca": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1aab1d48c6124526bee6c74892ecd953",
            "placeholder": "​",
            "style": "IPY_MODEL_598c6f37331e448889b175393a00deff",
            "value": "Downloading data files: 100%"
          }
        },
        "6a5d5ea44c0d4e4384b8956b48c34f14": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6ba50c3ec9e542d5a8d2f900fbcb8689": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6c0432c6cd8f4cae9aeb8c2870b39922": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_99b8f3882032404990f2b453111a63ba",
              "IPY_MODEL_6c21df0657ce486381ba2310c7fa0029",
              "IPY_MODEL_a20ac344a32340c785cd162fd0eb55b5"
            ],
            "layout": "IPY_MODEL_6ba50c3ec9e542d5a8d2f900fbcb8689"
          }
        },
        "6c21df0657ce486381ba2310c7fa0029": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2a75d45acfb44463af974acb7a1b0d8e",
            "max": 566426227,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_ae814b8e55454e0aadc48f7e595575ee",
            "value": 566426227
          }
        },
        "6ecbf23579324112981d4bce0c0ce369": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "6fde64f36dd84d8eae670efe95e2313a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2f1e652cfa514054abfda794bdfa61a5",
            "placeholder": "​",
            "style": "IPY_MODEL_b17f4731e8e44858889114c52b8f3dd4",
            "value": " 47.5M/47.5M [00:00&lt;00:00, 69.2MB/s]"
          }
        },
        "74d58314b1c449e0b7dec3bda8b653c2": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "782af098d37d4543a4a01ecfed4e5da2": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_62d8196169bb4a76a94c16d6f1ca61a6",
            "max": 46213747,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_0a16c7f37b9a4bbe9bfe7e737ea62801",
            "value": 46213747
          }
        },
        "7b0b45020d0f45288e2f0e4ceff22524": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_96ad4454539145aea6549995344160e3",
            "placeholder": "​",
            "style": "IPY_MODEL_acc3573e48f14dffb635f8632c6f6e74",
            "value": " 3/3 [00:24&lt;00:00,  5.68s/it]"
          }
        },
        "7d80528c4b1c48318756230b0174923c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_398d7d36bbd041fe81ec633e973fc504",
              "IPY_MODEL_d01ca0dd46ee4a4daeeee92214bc07d1",
              "IPY_MODEL_6028a5e08f8641f5b8e8182e50f55419"
            ],
            "layout": "IPY_MODEL_567646442eb940b09456328d49248945"
          }
        },
        "81cbe1842465400cba02a46309d98064": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_134fa83980e142bf82d5b97cfc10da69",
            "placeholder": "​",
            "style": "IPY_MODEL_35e4e99036894a2ba37d9ba23581a8ff",
            "value": " 6.42k/6.42k [00:00&lt;00:00, 208kB/s]"
          }
        },
        "835a1e675186490692e336da943d635d": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_95bcef02eb794979b7873b81021e4b40",
              "IPY_MODEL_d84324c4c3dc40fea04cc0df1539b4d8",
              "IPY_MODEL_a6213bcdbcb24b1694204e6baeb763d8"
            ],
            "layout": "IPY_MODEL_bc423d0878154adf940062660a8315ad"
          }
        },
        "84bed63c6597486ca6c39b9c0c4d20bd": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "8a64950ea896468da3d10f46e9718ec8": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_20d8df0ee0a4442582686846757bcc7f",
            "max": 3,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_fa4e50aaf53d4edfb9ca3046cbada2db",
            "value": 3
          }
        },
        "8b7b2b9489ae49049befae848d0fb1a1": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "8e396041a4fd4e6db02410a09b7556c7": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_4cd1dc71c80b401da19ed52ef5148d60",
              "IPY_MODEL_1f23a95e292249a19055f70cda2622a3",
              "IPY_MODEL_69412c7605ec48859baef6f31c91c520"
            ],
            "layout": "IPY_MODEL_ee64a46b61454949ade4177c059bcf61"
          }
        },
        "926cbb119ea745cf9e344c0e50b75f62": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "942897aa22214745adee56d2c54447e9": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "954f3cfcb5d44dddbf1d4db9e99c0d70": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "95bcef02eb794979b7873b81021e4b40": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0a1744c363f640b5b8939f32f6e72922",
            "placeholder": "​",
            "style": "IPY_MODEL_bbd2db64988147f1a4f8856d890bb4c7",
            "value": "Downloading readme: 100%"
          }
        },
        "96ad4454539145aea6549995344160e3": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "988055b44cf9411e8d45d8a4185fef07": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_29cab51fec0b4dacaa8f829ff217c839",
            "placeholder": "​",
            "style": "IPY_MODEL_6ecbf23579324112981d4bce0c0ce369",
            "value": "Downloading data: 100%"
          }
        },
        "99b8f3882032404990f2b453111a63ba": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a298045042014c88939a9fe785fccda8",
            "placeholder": "​",
            "style": "IPY_MODEL_84bed63c6597486ca6c39b9c0c4d20bd",
            "value": "Downloading data: 100%"
          }
        },
        "9a733957f8fc446fbde053ba87289c71": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9d4f14862b704d9bb53d9e74365d14ac": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e3dd508496f44171b0d91aca0e8b16a5",
            "placeholder": "​",
            "style": "IPY_MODEL_dc8077859eec4261a28d708ab06a1008",
            "value": "Generating train split: 100%"
          }
        },
        "9f07fa213de247dabcd3f4213d35a97e": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d6a78aabcca74314a5b4bb98f350693a",
            "placeholder": "​",
            "style": "IPY_MODEL_954f3cfcb5d44dddbf1d4db9e99c0d70",
            "value": "Downloading data: 100%"
          }
        },
        "a105ef4f1f754c44b7bc0ec8edf0c0cc": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a20ac344a32340c785cd162fd0eb55b5": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ff4bcec9c18e4f04b1f7898daaffeb1a",
            "placeholder": "​",
            "style": "IPY_MODEL_e9c64a790d684c9eaf7f49eea003f43c",
            "value": " 566M/566M [00:22&lt;00:00, 69.9MB/s]"
          }
        },
        "a298045042014c88939a9fe785fccda8": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a5effa5d67534fb4be2e3320c1fb2b9f": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "a6213bcdbcb24b1694204e6baeb763d8": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_43ed7af1d9c84ac8a6ec2195e48e60eb",
            "placeholder": "​",
            "style": "IPY_MODEL_ac6822762e6b46bd862d6f923aeb4437",
            "value": " 9.19k/9.19k [00:00&lt;00:00, 402kB/s]"
          }
        },
        "ac6822762e6b46bd862d6f923aeb4437": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "acc3573e48f14dffb635f8632c6f6e74": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ad21e20d1f1b4b6ea74fdab03af8acdb": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ae814b8e55454e0aadc48f7e595575ee": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "b17f4731e8e44858889114c52b8f3dd4": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ba2b4d7ecdbc4512acf180d8a0d602e8": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bbd2db64988147f1a4f8856d890bb4c7": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "bc423d0878154adf940062660a8315ad": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c7d10443100e49d28266aef9f644ed47": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d01ca0dd46ee4a4daeeee92214bc07d1": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f048b50740e94bd8805c142eac1673b6",
            "max": 7405,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_a5effa5d67534fb4be2e3320c1fb2b9f",
            "value": 7405
          }
        },
        "d50f5eafe90c4684bbdd96569b8ff247": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_9f07fa213de247dabcd3f4213d35a97e",
              "IPY_MODEL_782af098d37d4543a4a01ecfed4e5da2",
              "IPY_MODEL_037b10adf9724e058c6468f4ca74ed86"
            ],
            "layout": "IPY_MODEL_c7d10443100e49d28266aef9f644ed47"
          }
        },
        "d6a78aabcca74314a5b4bb98f350693a": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d84324c4c3dc40fea04cc0df1539b4d8": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ba2b4d7ecdbc4512acf180d8a0d602e8",
            "max": 9193,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_1bb155669e2a441d8992030e8aaa84a9",
            "value": 9193
          }
        },
        "dba7ce7c756d468b94d0bc8f4815ab6f": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "dc8077859eec4261a28d708ab06a1008": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "de6507e9f45741218bf31a9af728b2bf": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e16044d880174c49af557bd12789493f": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_531c380db44a404ebb168648ea77c3f4",
              "IPY_MODEL_e8a46c6ac8a54fdbb44b2c8914552e64",
              "IPY_MODEL_81cbe1842465400cba02a46309d98064"
            ],
            "layout": "IPY_MODEL_f8ada809ecdb417ebc8214d860dd6552"
          }
        },
        "e2f3a3377ad64a4cb9f3a42c1ac97344": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e3dd508496f44171b0d91aca0e8b16a5": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e8a46c6ac8a54fdbb44b2c8914552e64": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2d47a6a66054438d8e9a3c5e5767c056",
            "max": 6422,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_3b57d0a9768f4befb7509581289035ad",
            "value": 6422
          }
        },
        "e9935b60c48d46469904492e20f9c2e8": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e9c64a790d684c9eaf7f49eea003f43c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ee64a46b61454949ade4177c059bcf61": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f048b50740e94bd8805c142eac1673b6": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f120c447645446e4a04791b97ce9ae93": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_5cfebfd6308349038f9cd7ad5bc00fe5",
            "placeholder": "​",
            "style": "IPY_MODEL_56b80fc45dd247deadceffe17557f0f9",
            "value": " 90447/90447 [00:41&lt;00:00, 2721.55 examples/s]"
          }
        },
        "f8ada809ecdb417ebc8214d860dd6552": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "fa4e50aaf53d4edfb9ca3046cbada2db": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "ff4bcec9c18e4f04b1f7898daaffeb1a": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
